unity unet multiplayer with oculus rift vr - having the oculus only control the local player

I am loving the new unet features and it's really easy to setup a multiplayer environment.

I have a function to detect if the player is the local player, and if not, disable all movement so only the local player is being controlled. Works great.

However, when I have the Oculus Rift Headset on and VR enabled, the rotation/movement of the headset controls the camera for all of the connected clients. I cannot figure out a way to set this so it only controls the local player.

Anyone have experience with this before that can shed some light on the subject for me?

I am using all the latest builds of everything - unity 5.2.1.f1, latest oculus sdk, etc...

Thanks!! -Jesse