Separating Networked Objects into Rooms

I was wondering what the best method of separating clients is with the default Unity networking. My game is kind of like Monster Hunter in that there is a lobby that all clients sit in, then any number of them can move to a separate "room". So I need some method of ignoring RPCs from one to the other.

My initial thought was that this is what the "network groups" are for. But after building it out, I realized that there's no good way for a client to ignore RPCs from a different group, and that the group id is more meant as a way of collecting the buffered RPCs together to clear them when needed.

Is there any way to do this with the groups, or am I missing something else entirely?