Online Multiplayer Oculus Rift Movement issues

Hi! Some friends and I are trying to set up a networked oculus rift game, but after following a simple tutorial on networking, we're having some issues.

We set up a Network Manager object and made the necessary changes to the OVRPlayerController to network-proof it.

The issue though is that when multiple people are in the server, the only thing that seems to move is the meshfilter we stuck on the playercontroller.

Another strange occurrence is that host's screen will madly flicker as long as the client is input. Between the flickers, the host can see what is usually on his screen (the simple plane and his somehow disjointed mesh filter), and then he sees the plane from another angle (usually slightly higher?), where he sees the mesh filter of the other player moving around. As soon as the client takes his finger off the key, then the flicker stops and the host sees his normal view all the time.

Our network solution is pretty basic: Using standard Unity Networking; If no server for our game, set up server, spawn player Other user finds server, joins, spawns Serialize positions and velocities and send them between the two and lerp to make it smooth.

We've tried putting a network view on the camera object as well for the player prefab, thinking that might be why the player sits still, but that was not it.

Does anyone have any experience with networking in Unity? I don't know if this is an oculus thing or a character controller thing, but we're pretty stumped.

EDIT:

We've managed to fix everything by switching to Photon, but now whenever the other person joins, the first oculus seems to to 'snap' to their object and both players share the same camera position, even thought WASD movement still works for their respective controllers.

Any help is appreciated, and thanks so much in advance.