Gravity by raycasting almost working

I have been working on a project in Unity, in which I was using a raycast to detect the normal of the ground below the player. The player would then have it's up rotation set to the hit.normal of the raycast. This worked perfectly for the top half of a sphere world. When you go to the bottom half of the sphere, the player would start jerking around when the normal value changed. If you ever reached the bottom of the sphere, the player would jerk around uncontrollably, so that you couldn't move in the direction that you wanted to.

The code used for the player movement:

using UnityEngine; using System.Collections;

public class GravityRaycasting : MonoBehaviour {

 Rigidbody myBody;

 public RaycastHit rayHit;

 // Use this for initialization
 void Start () {

     myBody = this.GetComponent <Rigidbody> ();

 }
 
 // Update is called once per frame
 void FixedUpdate () {

     Physics.Raycast (transform.position, -transform.up, out rayHit, Mathf.Infinity);

     transform.up = Vector3.Lerp (transform.up, rayHit.normal, 0.25f);

     myBody.AddForce (-transform.up * 50f);

     Debug.DrawLine (transform.position, rayHit.point, Color.green);

     myBody.AddForce (Camera.main.transform.forward * Input.GetAxisRaw ("Vertical") * Time.deltaTime * 560f);
     myBody.AddForce (Camera.main.transform.right * Input.GetAxisRaw ("Horizontal") * Time.deltaTime * 560f);

 }

}

Here is the mouse look script, but I doubt that this could be causing the problem:

using UnityEngine; using System.Collections;

public class MouseLook : MonoBehaviour {

 public float lookSensitivity = 5f;
 
 public float yRotation;
 public float xRotation;
 
 float currentYRotation;
 float currentXRotation;
 
 public float yRotationV;
 public float xRotationV;
 
 public float lookSmoothDamp = 0.1f;
 
 public bool UseYAxis = true;
 public bool UseXAxis = true;

 public GameObject playerObject;

 void FixedUpdate () {

     if (UseYAxis) {

         xRotation = Mathf.Clamp (xRotation, -90f, 90f);
     }
 }
 
 void Update () {

     if (UseXAxis) {
         
         yRotation += Input.GetAxis ("Mouse X") * lookSensitivity;

         currentYRotation = Mathf.SmoothDamp (currentYRotation, yRotation, ref yRotationV, lookSmoothDamp);
     }
     
     if (UseYAxis) {
         
         xRotation -= Input.GetAxis ("Mouse Y") * lookSensitivity;
         
         currentXRotation = Mathf.SmoothDamp (currentXRotation, xRotation, ref xRotationV, lookSmoothDamp);
     }

     transform.localRotation = Quaternion.Euler (currentXRotation, currentYRotation, 0f);

 }

}

Any help would be appreciated.

Thanks.