Online, Possible Memory Leaks with RPC

I am working on a multiplayer game with a few classes that are synchronized to enable movement for the player, there are very few variables that are synchronized in these classes and I send RPC's from time to time to instantiate and kill players. Some, not all of the RPC's are buffered.

The problem I am facing is that the performance degrades over a period of time. So, on a Mac Book Pro, it start out at around 80-90 fps, stays like that for a good 20-25mins and then slowly starts slowing down to the point that after an hour to an hour and a half its at 20FPS!!

So, the graphics rendering and physics is not the bottle neck, since it starts out pretty well performs well for 20-30mins. I am guessing that it has something to do with the memory leaks or the RPC's buffer. But how can I figure out exactly what? According to the profiler the total allocated memory goes up from around .82GB to .85Gb after an hour.

Any good reads on memory leaks for unity? Any good reads on performance degradation over time?