Socket networking performance tips sought

I have a fairly simple app. It's a client of a server that provides data frames between 5 and 8k in size. This is running locally (same machine, or two machines on the same router) at the moment, with intent to go wider.

The problem is, when I'm connected, my frame rate is horrible, like 3 to 6 FPS. When not connected, well above 30 FPS. I would not have thought that grabbing 5k chunks via a socket would have such a huge impact. I must be doing something wrong.

From what I can tell, I'm not blocking. I use an async callback to gather the incoming data, which usually takes two or three callbacks to get, before stuffing that in a final buffer to deliver to my app.

I'm not a networking guru, but I know some of you are. Is this just a limit of Mono? Or .NET 2.0? Is there a more efficient way of getting data of this size across a network? I don't think this is like a noisy connection, even localhost runs this slow. Would it help to put the socket stuff in an assembly? Or a DLL?