Remove jitter from interpolation of multiplayer networking (C#, Smartfox Server)

I've finally finished making a "simple" multiplayer networking project using C# and Smartfox Server, including interpolation (I used the code included in the Networking example and then modified it to work with SFS). This works fairly well, and the movement is quite smooth, but there is still occasional jitter.

It appears that the jitters happen when the latency increases (as would be expected), so that the 100ms of interpolation has happened, but the next position packet hasn't arrived yet. I don't know if I'm just too brain dead from working on this all day, but I can't seem to figure out a way to remove the jitter. Currently I get the timestamp in ms from the server and then increment it on the client side, but should I be sending the timestamp with every single update from the server (this seems like it would add even more unneeded latency)? I'm not looking for code necessarily, but rather just a conceptual explanation of how I could go about eliminating the jitter.