Unity hinge joint jitter when target gameObject parent is moving

I have a player car game object that moves forward with the following script:

     float step = speed * Time.deltaTime;
     transform.position = Vector3.MoveTowards(transform.position, targetPosition, step);

Inside this game object I have a child called Body and inside it another child called Neck Joint.

I would like to make a Bobble Head attached to my car object, more specifically to the Neck Joint. I have attached a head object with a hinge joint that works very well when the car is steering left/right (or rather when I swing the Body's local position left/right). The hinge joint's axis is in the z-direction and it has a spring setting.

My problem is when I move the parent player object with the above formula, the head seems extremely jittery in the z-axis. The left right swing works fine as per the hinge joint axis, but when my movement is in the direction of the hinge joint axis, I get really weird jitter as though the hinge joint is trying to catch up to the object.

I tried setting the physics iteration count to 100 and still the same issue.

Sorry if I am not clear, ask away if you didn't understand anything. And thanks for the help!