Unity networking non-local player and camera disabling bug

I'm working on a multiplayer top-down shooter, and right now I'm setting up the basic networking. I ran in to a problem when entering the scene. I want to disable the player controller script and the camera and audio listener of all non-local players from any given players side (I hope this is good enough of an explanation). Here is what happens:

In the PlayerController script I instantiate the player's camera like this:

   PlayerCam = Instantiate(PlayerCam, transform.position, transform.rotation) as Camera;
   PlayerCam.GetComponent<CameraControl>().target = transform;

The second line sets the Camera's target to be the Player himself, so that the camera can follow him by position only, no rotation (that's the function of the CameraControl script.

Then I have the OnlineSetup script, which should be disabling all non-local players (i.e. PlayerController scripts) and cameras (i.e. CameraControl scripts) by doing this in the Start() of the OnlineSetup script:

 if (!isLocalPlayer)
         {
             for (int i = 0; i < ToDisable.Length; i++)
             {
                 ToDisable[i].enabled = false;
             }
         }
         else
         {
             SceneCamera = Camera.main;
             if(SceneCamera != null)
             {
                 SceneCamera.gameObject.SetActive(false);
             }
         }

And in the array of behaviors I have PlayerControler, Camera and Audio Listener. The idea is that this disables these behaviors if they belong to a player other than the local one, so that they don't read the local players input.

However, I've found that this is probably not the way to do it, since this disables the local player camera, as well as disabling the client and his camera on his side, which is not what i want. What I want to know is what is the best way to go about this. The thing is, I can't make the camera a child of the player, like wone would do in an FPS or a regular TPS, since the functionality of the camera would break.

I'm basically looking for a way for the player to controll only his PlayerController(and subsequently his own camera) and not those of the others that connect to the game.