How to walk around a 3D sphere?

I'm making a small 3D space simulator. Currently I have a character that I want to have walking around a planet. The problem is that the character acts as if I am rotating him around the planet, while what I am trying to do is using the arrows/WASD to move around and have my own physics functions fix the rest like gravity and keeping the character standing upright. I think the problem lies in my attempt to keep the character upright...

How I do this is using a (invisible) sphere as a base for the player. This sphere has an active collider and a rigidbody. It is oriented with its transform.forward pointing toward the center of the Planet. The "actual" player is a capsule with active collider (no rigidbody) inside this sphere as its child, that has its forward pointing on a 90 degree angle from the sphere (270 on the X-axis in the inspection view). The character script it attached to this capsule.

     myTransform = transform.parent;

     void Update(){
         //gravity doing it's job GravitationalForce() returns a constant for now
         Vector3 gravityDirection = currentPlanet.transform.position - transform.position;
         myTransform.position += gravityDirection.normalized * currentPlanet.GravitationalForce(weight) * Time.deltaTime;


         //rotation kept 90 degrees from gravityDirection
         //invisible sphere around people, pointing downwards topward center of planet
         myTransform.rotation = Quaternion.Slerp(myTransform.rotation, Quaternion.LookRotation(gravityDirection), 100.0f);

         //movement
         Vector3 movementDirection = new Vector3(0.0f, 0.0f, 0.0f);
         movementDirection += transform.forward * Input.GetAxis("Vertical") + transform.right * Input.GetAxis("Horizontal");
         movementDirection.Normalize();
         myTransform.position += movementDirection * walkingSpeed * Time.deltaTime;
     }

My current hypothesis is that when the outer sphere rotates to lookAt the planet's center it rotates around multiple axes, making the player rotate with it, producing the RotateAround() effect. Could anyone help me get around this problem? Or if my hypothesis is wrong, tell me what the real problem is and help me fix that?

Extra notes: I prefer to make my own physics functions over using Unity's as I'm trying to teach myself game physics. Using Unity's physics ruins this exercise.