First person camera judder with rigidbody based characters

I've put together a rigidbody based first person controller system for the project I'm working on. Basically the issue that I have is that I get awful levels of judder when rotating the camera whilst moving at the same time moving the player. Here is a video of it for context. The judder is shown first, followed by what it should be like if I could get it to work properly:

https://drive.google.com/file/d/0BwmJXNLk6lQILTBiYlVfV0QyUUE/view?usp=sharing

This issue has actually been posted about multiple times, but all of the answers that I've seen either haven't worked for my situation or have fixes that at least from what I can tell have unnecessary side effects. So far I've tried and failed with:

• Adding Time.deltaTime to some of the calculations in the script below.

• Recording the deltaTime inside fixed update as a variable and using that as the smooth time in the SmoothDamp function.

• Setting the rigidbody to interpolate, extrapolate and even changing the collision detection to continuous/ continuous dynamic.

Interestingly this is what has worked:

• Moving the camera input from LateUpdate or Update to FixedUpdate. This actually fixes the problem (probably as they're both in sync?), however I'm aware that FixedUpdate should only be used for physics work. Coupled with this causing the camera rotation speed to depend on the fixed time step, doing this isn't really practical if I have differing frame rates used on an end user's machine.

• Decreasing the fixed time step to 1/60 (0.0166666) completely eradicates the problem. However this is at the expense of more physics calculations and causes the same issue mentioned above. That and it's limited to 60FPS.

Here is my camera script to get more context. I can post up the rigidbody movement controller if need be. Any other ideas would be appreciated for fixing this as I'm out of luck at this point:

https://drive.google.com/file/d/0BwmJXNLk6lQILTBiYlVfV0QyUUE/view?usp=sharing

 using UnityEngine;
 using System.Collections;
 
 /// <summary>
 /// Handles our first person camera motion
 /// </summary>
 [RequireComponent(typeof(CameraUtilities))]
 public class FirstPersonCameraController : MonoBehaviour
 {
     #region Variables
     public Camera camera;
     public Rigidbody rigidbody;
     public float clampValue = 90f;
     public float lookSensitivity = 5f;
     public float smoothingInertia = 0.05f;
     public Vector2 mouseInput;
     public Vector2 mouseAbsolute;
     public Vector2 smoothedRotation;
     public Vector2 currentSmoothedVelocity;
     private Quaternion xRotation;
     private Quaternion yRotation;
     private Quaternion cameraStartDirection;
     private Quaternion rigidbodyStartDirection;
     #endregion
 
     #region Singleton
     private static FirstPersonCameraController _instance;
 
     public static FirstPersonCameraController instance
     {
         get
         {
             if (_instance == null)
             {
                 _instance = new FirstPersonCameraController();
             }
             return _instance;
         }
     }
     #endregion
 
     /// <summary>
     /// Processes input from the mouse to calculate the camera movement
     /// </summary>
     private void CameraInput()
     {
 
         // get raw mouse input 
         mouseInput = new Vector2(Input.GetAxisRaw("Mouse X"), (Input.GetAxisRaw("Mouse Y")));
         mouseInput = Vector2.Scale(mouseInput, new Vector2(lookSensitivity,
             lookSensitivity));
 
         // smooth the mouse input and increment
         smoothedRotation.x = Mathf.SmoothDamp(smoothedRotation.x, mouseInput.x,
             ref currentSmoothedVelocity.x, smoothingInertia);
         smoothedRotation.y = Mathf.SmoothDamp(smoothedRotation.y, mouseInput.y,
             ref currentSmoothedVelocity.y, smoothingInertia);
         mouseAbsolute += smoothedRotation;
 
         // clamp the mouse data y axis so we can't back flip
         mouseAbsolute.y = Mathf.Clamp(mouseAbsolute.y, -clampValue, clampValue);
 
         // calculate our finished axis rotations
         xRotation = Quaternion.AngleAxis(-mouseAbsolute.y, Vector3.right);
         yRotation = Quaternion.AngleAxis(mouseAbsolute.x, rigidbody.transform.up);
 
         // apply axis rotations
         camera.transform.localRotation = cameraStartDirection * xRotation;
         rigidbody.transform.localRotation = yRotation * rigidbodyStartDirection;
 
     }
 
     private void Awake()
     {
         if (_instance == null)
         {
             _instance = this;
         }
         else
         {
             Destroy(this);
         }
     }
 
     private void Start()
     {
         // get the rigidbody and camera for rotating 
         camera = CameraUtilities.GetMainCamera();
         rigidbody = Player.instance.GetRigidBody();
 
         // get our starting directions
         cameraStartDirection = camera.transform.localRotation;
         rigidbodyStartDirection = rigidbody.transform.localRotation;
 
         // clear them all to zero 
         currentSmoothedVelocity = smoothedRotation = mouseAbsolute = Vector3.zero;
 
     }
 
     /// <summary>
     /// Switching this to FixedUpdate fixes judder
     /// </summary>
     private void Update()
     {
         CameraInput();
     }
 }