Networked nonplayer objects only effected by the client hosting the server?

I have just set up a billiards based game that I am attempting to network. I am able to host a game and connect to it via multiple clients, however, the clients seem to be completely unable to effect the server in any way. The server hosted client is able to interact normally and the results are displayed to all clients. I feel like it is just a tiny detail I'm missing but it's driving me crazy. I have tried seemingly every variation of NetworkIdentity and NetworkTransform parameters for my "Ball" object.

Here is a link to a video demonstration of the issue I am having: https://youtu.be/ekc4UmLYQ_k

I have 15 "Ball" GameObjects with default NetworkIdentity. They are instantiated and spawned through OnStartServer(). This object is also registerred spawnable with the NetworkManager.

I have 2-4(number of players, 2 ideally, 3 shown in video for testing) "TestPlayer" Objects which are spawned at two spawn points automatically by the NetworkManager. They have local NetworkIdentity.

I planned to have players take turns in billiards and let Unity handle the collisions, it looks great from the server side, just unplayable by clients.

My current observation is that the server is setting the balls back to their default position when the client collides with them, but not doing this when the server client does.

EDIT*** I set my NetworkTransform update rate to 0 on my player objects and changed their movement to be done through a command CmdMovePlayer, this does everything that I want EXCEPT the remote client player avatar does not update in its own client, only on the server's client. However if I add an update rate to the Player object, My movement stops working on remote clients.

Is there a way to control remote client player movement via Commands while keeping a NetworkTransform component with a positive refresh rate?