addRelativeforce problem after rotation

So I'm trying to make an object move towards a sphere using addRelativeforce, which works great as long as the object doesn't rotate, as soon as it rotates, it starts to act wobbly and eventually flies off as if the force is being applied from some other direction.

I'm trying to make something physics based in which the objects can rotate around the 'planet' and always fall towards it.

I had this working before but its been some years and I lost the files so I can't check what I did differently back then.

This is the code I'm using:

 using System.Collections;
 using System.Collections.Generic;
 using UnityEngine;
 
 public class PlanetBehavior : MonoBehaviour {
 
     public GameObject planet;
     public float Gravity_range = 10f;
     public float Gravity_force = 1f;
 
     // Use this for initialization
     public void Start()
     {
 
     }
 
     // Update is called once per frame
     void Update()
     {
 
     }
 
     void FixedUpdate()
     {
         Planet_grav();
     }
 
     public void Planet_grav()
     {
         Collider[] objects_in_range = Physics.OverlapSphere(planet.transform.position, Gravity_range);
         foreach (Collider hit in objects_in_range)
         {
             if (hit.tag == "gravityaffected")
             {
                 Rigidbody hitbody = hit.GetComponent<Rigidbody>();
                 hitbody.AddRelativeForce((planet.transform.position - hit.transform.position).normalized * Gravity_force);
             }
         }
     }
 }