Very jerky camera movement with rigidbody

I have a rigidbody that is flicked on the iphone and is sent flying. What happens is the rigidbody rotates like crazy (which is what I want) and the camera follows the rigidbody on its z axis. When the rigidbody rotates, the z value goes up and down really quick, which causes the camera to move back and forth very quick and making it very jerky (gives me a headache just looking at it).

What I tried:

I made the camera only move with the rigidbody if the z value was larger then the last z value. So now I don't have the camera that shakes back and forth but I have a camera that stops and and then goes while waiting for the rigidbody to rotate and increase the z value. I tried lerping as well to see if it would smooth it out. (It didn't still jerky).

I'm not sure if you will understand this correctly, but if you guys need a video to see what I mean I will post one.

I hope you guys can help me with this problem. It has been giving me a headache the past 2 days. Literally!