Multiplayer jitter on client

I'm working on a multiplayer hockey game with an authoritative server, but I'm having a strange jitter on the clients that I can't seem to track down. On the server it looks (relatively) smooth, but on the client it looks like this:

I don't have any prediction/compensation yet because I didn't want to cover up the jitter so the general jerkiness is understandable, but that shouldn't cause it to lurch forward and back again every few frames like that.

I'm sending the input from clients to the server and running my movement code there so there shouldn't be any conflict there.

I thought it might be that the client was calculating physics based on the velocity coming back from the server and it was conflicting with the updates coming from the server, but if I Destroy the rigidbody in Start on the clients, it doesn't make any difference. Now that I think about it, I don't have a NetworkView monitoring a rigidbody or anything anyway, so that can't be it.

I also thought it might be an issue of packets arriving out of order, but changing the NetworkView to Reliable Delta Compressed should prevent that and it also makes no difference. I also tried printing timestamps and they were all in order.

I'm out of good ideas what could be causing this. Help?

Edit: Here's another gif that demonstrates it better. I turned the sendrate down so you can more clearly see the time between updates from the server that I could smooth out with interpolation and the weird inconsistent jerk forward then back on some frames that I don't know how to fix.