Networking A first person camera with projectiles

I have a first person camera setup and working fine on unity in single player as well as if the instance is hosting, but it has problems with projectiles when the instance is a client rather than a server.

When I host a server, I can look around and when I click, a projectile is summoned in the direction I am looking. If I look up it goes up. However when I connect with a client, the camera views fine, but the projectile is not summoned going the correct direction, it follows my y rotation (top down rotation) but not my x rotation (looking up and down). So that if I spin my camera it will fire forward, but if i look up or down it still fires forward as if I didn't look up or down.

Code for firing projectile (this is [command] and is run by a script when the local player clickes):

         GameObject proj = Instantiate(projectile, cam.transform.position + cam.transform.forward , cam.transform.rotation);
         proj.GetComponent<Rigidbody>().velocity = cam.transform.forward * 10;
 
         NetworkServer.Spawn(proj);
 
         Destroy(proj, 10);

I have verified "cam" is the correct camera and not another one, as well as the fact that I am syncing all of my rotations (XYZ) to the server in the network identity and transformation scripts.

My player is a first person character with its own network identity and transforms scripts.