Moving in VR using planetoid-like center of gravity

I'm new to unity and VR and I've found the concept of the environment set on a small planet one that I would like to focus on. Maybe going a step further and creating a "hollow planet" game

I've got the basics for setting the graviity to a fixed point however I would like to implement VR gaze-based movement direction but the scripts of VRTK seems unresponsive in my setting. Can use a separate touchpad-only movement no problem but I'm interested to see if VRTK could work

The scripts I'm using for the gravity are these:

public class GravityAttractor : MonoBehaviour {

 public float gravity = -9.81f;

 // Set the center of the object as the center of gravity that affect objects
 public void Attract(Transform body) {

     // The gravity direction perpendicular to the center of gravity
     Vector3 gravityUp = (body.position - transform.position).normalized;
     // The "up" position of objects affected by gravity
     Vector3 bodyUp = body.up;

     // Add gravity pull to objects
     body.GetComponent<Rigidbody>().AddForce(gravityUp * gravity);
     // Keep the objects oriented to the center of gravity
     Quaternion targetRotation = Quaternion.FromToRotation(bodyUp, gravityUp) * body.rotation;
     body.rotation = Quaternion.Slerp(body.rotation, targetRotation, 50 * Time.deltaTime);

 }

}

public class GravityBody : MonoBehaviour {

 GravityAttractor attractor;
 private Transform myTransform;

 void Awake() {

     // Attract and orient the rigibodies to the center of gravity (the planet selected automatically)
     attractor = GameObject.FindGameObjectWithTag("planet").GetComponent<GravityAttractor>();
     GetComponent<Rigidbody>().useGravity = false;
     GetComponent<Rigidbody>().constraints = RigidbodyConstraints.FreezeRotation;

     myTransform = transform;

 }
 
 // Update is called once per frame
 void FixedUpdate () {

     attractor.Attract(myTransform);
         
 }

}

There's also a weird bug. At alternate times when I start the game the character flips on his own, no idea what's causing it

Any suggestions?