Question about having a Rigidbody follow another GameObject via forces.

disclaimer: I'm extremely new to Unity (second week)

Hey guys, as an intro to the physics system of Unity, i'm making a sword fighting game. I've got most systems i'm trying to build working, but i'm struggling with something that seems quite basic.

I'm trying to make all of the swordfighting happen via rigidbodies, so blocking and hitting feels real (or at least interesting). The first step of this is to make the "sword" object follow my player's head (first person, head rotates around, camera attached to head). The way i did this at first was very simple, by just nesting the sword as a child of the head, or just moving the transform in code. However, i want it to be possible to move the sword (via an external interaction, like another sword hitting it) at any time. So, i decided to attach a Rigidbody to the sword, and try to make it follow the player using forces.

The system i wrote almost does this, by calculating the difference in position and rotation and applying a force and torque to move towards said position and rotation. I'm using Rigidbody's built-in drag to slow it down. This is the system i'm using right now:

 [RequireComponent(typeof(Rigidbody))]
 public class SwordController : MonoBehaviour {
 
   public Transform followObject;
 
   public float followSpeed = 20000f;
   public float angularFollowSpeed = 20000f;
   public float drag = 10f;
   public float angularDrag = 10f;
 
   private Rigidbody body;
 
   private void Start() {
     body = GetComponent<Rigidbody>();
     body.drag = drag;
     body.angularDrag = angularDrag;
     body.useGravity = false; // hand is holding, no need to use gravity.
   }
 
   private void FixedUpdate() {
     // follow the followObject using force
     // this requires drag on the rigidbody (not to go back and forth like crazy), which is set in Start
     Vector3 difference = followObject.position - transform.position;
     if (difference.sqrMagnitude > float.Epsilon) {
       body.AddForce(difference * followSpeed * Time.deltaTime, ForceMode.Force);
     }
 
     // follow the rotation of followObject using force
     // also requires angular drag on the rigidbody not to go back and forth like crazy.
     Quaternion rotationDifference = (Quaternion.Inverse(transform.rotation) * followObject.rotation);
     Vector3 rotationDifferenceVector = new Vector3(rotationDifference.x, rotationDifference.y, rotationDifference.z);
     if (rotationDifferenceVector.sqrMagnitude > float.Epsilon) {
       body.AddTorque(rotationDifferenceVector * Time.deltaTime * angularFollowSpeed, ForceMode.Force);
     }
   }
 }

However, this doesn't really work. It works for the position, albeit the massive drag and followSpeed numbers i'm using make me feel like i'm doing this in a wrong way. The rotation doesn't work for certain "quarters" of rotation, where it starts doing this weird orbiting thing, also, it's extremely jittery, to an unusable extent.

I would love to hear if i'm approaching this in a very wrong way, or if there's a nice (or not so nice) way to fix my rotation problem.

Thanks!