UNET 5.3 Shooting Client/Server Sync issue

I'm trying to create a UNET 5.3 game for 2 players. One player is the Server/Host, while the other is the Remote Client.

My issue here is with shooting bullets.

Both players can shoot bullets, which are spawned as prefabs from the Player prefabs. It works fine when the Host player fires - the Remote Client sees the bullets fired by the Host just fine. The bullets either move off the play area (in which case the boundary destroys them), or hit a target and trigger the Server's collision/damage calculation stuff.

BUT when the Remote Client fires, it seems some (around half) of his bullets 'disappear' prematurely for the Host player, ESPECIALLY when the Remote Player object is moving around. And because the bullet's damage collision/damage logic are handled on the Server side, this means a lot of the bullets fired by the Remote Client simply don't hit!

Here is the code used (there are bits I have removed for simplicity), and the idea is this:

1. Local player triggers ClientFire() via input

2. ClientFire() runs on the local client. It instantly creates a local bullet object for the Remote Client, and gives it some initial parameters (like the bullet's owner for friendly-fire considerations). This local bullet is purely to give the Remote Client a feeling of instant response. The local bullet doesn't spawn for the Server/Host player, as he doesn't feel any response lag anyway.

3. ClientFire() triggers CmdFire() command on the Server. The command uses a NetworkSpawn to create a bullet in the Network, and gives it similar initial params as above.

4. CmdFire() finally calls a ClientRpc(). Here the Network's bullet is destroyed for the Remote Client, as he is only interested in seeing his local bullet (I know it'd be better to show him the Network one and destroy the local, not vice versa... but whatever).

Also note that the bullet prefab has a NetworkIdentity and NetworkTransform to sync its transform between Server/Clients.

 void Update () {

     if (!isLocalPlayer) {
         return;
     }
         
     if (Input.GetKey (KeyCode.LeftControl)) {
         ClientFire ();

     }
 }

 [ClientCallback]
 void ClientFire () {

     if (isServer)
         Debug.Log ("Server: I don't need a local copy because my bullets spawn instantly...");
     else {
         Debug.Log ("Remote: I do need a local copy because my bullets spawn on server after a while...");
         GameObject o = (GameObject)Instantiate (bullet, weaponSlot.position, weaponSlot.rotation);
         o.GetComponent<CollisionController> ().InitParams (gameObject);
     }

     CmdFire ();
 }

 [Command]
 void CmdFire() {

     GameObject o = (GameObject) Instantiate (bullet, weaponSlot.position, weaponSlot.rotation );
     o.GetComponent<CollisionController> ().InitParams (gameObject);
     NetworkServer.Spawn (o);

     RpcSetParamsForFiring (o);
         
 }

 [ClientRpc]
 void RpcSetParamsForFiring(GameObject o) {

     if (!isServer) {
         if (isLocalPlayer) {
             Debug.Log ("Remote: I don't need this server object because I have a local copy...");
             Destroy (o);
         } else {
             Debug.Log ("Remote: I do need this server object because I don't have a local copy...");
             o.GetComponent<CollisionController> ().InitParams (gameObject);
         }
     }
 }

I can't figure out what is causing this sync issue. Please help!