Changing player object UNET

Hello, people

I'm making a little game where players can change what object they're cotrolling. Like they start with a ball and they can roll it using WASD, and when they collide with some other ball tagged "PlayerBody", they will have control on that ball and lose it on old one.

Here's what I do (PlayerBodyScript.cs inheriting from NetworkBehaviour):

 private void OnCollisionEnter(Collision col) {
         if (tag == "Player") {
             // ... some code about jump landing
 
             if (col.gameObject.tag == "PlayerBody" && plyLastUpdated + 3f < Time.time){
                 // set new body if collided with PlayerBody
                 col.gameObject.GetComponent<PlayerBodyScript>().color = color;
 
                 if (isLocalPlayer){
                     // local player updates the camera
                     col.gameObject.GetComponent<PlayerBodyScript>().cam = GameObject.Find("Main Camera").transform;
                     GameObject.Find("Main Camera").GetComponent<CameraScript>().UpdateTarget(col.gameObject);
                 }
 
                 if (isServer) NetworkServer.ReplacePlayerForConnection(connectionToClient, col.gameObject, playerControllerId);
 
                 // unsetting player from this object
                 col.gameObject.GetComponent<PlayerBodyScript>().CmdChangeTagTo("Player");
                 CmdChangeTagTo("PlayerBody");
 
                 color = Color.white;
 
                 plyLastUpdated = Time.time;
             }
         }
     }

But every time I touch other PlayerBody I got this error and user takes control over both bodies (I do not want to destroy old one, but just "move" user to the other "body"):

I am wondering what I'm doing wrong? I'm kinda new in UNET and Unity overall.
Thank you for any help.