Move object along the curved surface. Problem moving to bottom hemisphere.

Hello everybody!

I know there has been a lot of questions on this. I've browsed through ton of topics. But I couldn't find the answer that would fit me.

What I'm trying to do is to move the object along the curved surfaces. And everything is just fine, until you reach a 90 degree point, when it starts to turn upside down. If placing it on the sphere, it just doesn't want to cross the equator. And if I force it to, it falls off.

Here's a short video:

https://www.youtube.com/watch?v=xLcjTbSiD_g

Here's the code:

     if (Physics.Raycast(ray.origin, ray.direction, out hit, 100, 1<<mask.value))
             {
                 delta = new Vector3 (0,Input.GetAxis("Mouse X")*1000*Time.deltaTime,0);
     
                 controller.transform.rotation = Quaternion.FromToRotation (transform.up, hit.normal) * transform.rotation;
                 controller.transform.Rotate(delta);
             }
     
     if (controller.isGrounded)
             {
                 moveDirection = new Vector3(Input.GetAxis("Horizontal"), 0, Input.GetAxis("Vertical"));
                 moveDirection = transform.TransformDirection(moveDirection);
                 moveDirection *= speed;
                 if (Input.GetButton("Jump"))
                     moveDirection += hit.normal*jumpSpeed;
             }
 
             moveDirection -= hit.normal*gravity * Time.deltaTime;
             controller.Move(moveDirection * Time.deltaTime);
         }

I use the RigidBody set to kinematic and a Character Controller on the object.

Yes, I've seen this Normal Walker and used this code as the start reference, however I do not get the part with the threshold and even having this code applied did not make it work.

I know that it has something to do with the local and global rotation. And that it probably happens due to the fact that the object rotates locally, the world Y Axis is still facing up. So when I reach the equator the Y Axis starts to face inside of the sphere. But I just do not know how to overcome it.

I've seen this and this but topics just seem to freeze when they come to the question I need :-). So please help, I'm really stuck with it.