[UNET] Only spawn certain server objects on local client?

So I have an interesting problem and I'm wondering if I can improve performance with UNET. In my game there are two sets of clients: O Clients and D Clients. Each of these teams of clients have their own objectives, and neither team needs to see or receive any information from the other team of clients.

However both teams of clients must be connected to the same server, so that the server can tally the scores of O Client and D Clients combined. Currently, on a local client on the O team I simply disable renders on all D client objects spawned by the server, and vice versa. However, when I use NetworkTransforms, or update SyncVars for a single client, the server sends that data over the network to both teams of clients, even the team that doesn't need this info. This is essentially doubling the amount of network traffic I have.

What I'm wondering is if there's a way for a local client on the D team to destroy all clients on the O team on their machine, so that the server knows not to send them any info about the O team. The other solution would be if there's a way to put a filter on syncvars and NetworkTransforms so that they only send updates to clients of the same team.

The desired solution is that a local client on the O team sends NetworkTransform data to the server, the server then sends that NetworkTransform data to all clients marked as on the O team, but doesn't send that data to any clients marked on the D Team.

Does anyone know if this is possible? Or have other suggestions which may help?