Raycast doen not work on network client

I have a networked game which is divided between clients and a server.

When I run the client in the editor, I can select all the characters via Raycast, no problem. However, when I build a standalone for windows, the chars are ignored by the raycast.

When I build the server standalone, the same chars are selectable (as well as in the editor). Is this an issue, because the objects are instantiated on the server? (Network.Instantiate)?

Any other ideas?

The characters have box colliders and are in the default layer. Everything is enables and looks good to me.