Multiplayer: Rigidbodies, character controllers or something else?!

I'm creating a multiplayer game, and I was curious as to the best way to implement players and movement - because I've ran into ridiculous issues with any route I go.

I know physics are tough and/or laggy to get right over a networked game, so I decided to first try the character controllers. My game takes place in interiors, and thus there are walls. With the character controller on my player, they can walk right through walls. I threw kinematic rigidbodies on the walls, after hearing that character controllers do have collision with rigid bodies... but I guess I read it wrong.

So I went with rigid bodies on the players. When the game starts, my player begins sliding around uncontrollably. This doesn't have to do with the movement scripts I have, the physics object just slides around when there is no input. I read some people having similar issues with the solution being "it's just the physics engine, deal with it." They recommended me mess with the drag, but then the player becomes unmovable! (I also changed the collider from a capsule to a cube, thinking it might have to do with that, but no luck.)

So basically, I want to know what is the best way to do movement with a multiplayer game, whether rigid bodies are better or simply changing transforms and positions of players (there is no need for physics in the game) through code - AND any help with collision using character controller would be fantastic!

Just one issue after another with this project :\

Thanks!