SplitScreen shooter with Viewmodels and PlayerModels without so many layers?

So I am trying to make a split screen fps where each player has a viewmodel and a playermodel. The viewmodel needs to only be visible to the current player and the playermodel only needs to be visible to other players.

The only way I can think of is modifying the cullingmask for the player so that their view model gets put into a layer that they can see and their model gets put into another layer that they can't. Because the layers are universal and not tied to a specific GameObject or camera this means I need to make 2 layers for each player so that other players can still see what the current player can not. I thought maybe I could assign the player model to the same layer as the other players but this creates a domino where each player couldn't see one of the players models.

My question is if there's any other way to do this without making 2 layers per player? Once again it's 2 layers because the player needs to be able to see their viewmodel and not their player model while other players need to see other players playermodels but not their viewmodels. Because this is a shared space I can't assign to one layer that nobody can't see because then nobody will be able to see.