Rigidbody rotating too slowly

So basically I "made" a script that controls the hand movement, and has proper collisions too. Most of it works, however, the rotation seems to have a limited speed, even if its speed is set to high values.

The speed of the rotation is slow, no matter what I change the multiplier to.

Here is the code:

 using System.Collections;
 using System.Collections.Generic;
 using UnityEngine;
 
 public class Hand: MonoBehaviour
 {
     // Physics Movement
     [SerializeField] private GameObject followObject;
     [SerializeField] private float followSpeed = 30f;
     [SerializeField] private float rotateSpeed = 100f;
     [SerializeField] private float teleportDistance = 0.75f;
     [SerializeField] private Vector3 positionOffset;
     [SerializeField] private Vector3 rotationOffset;
     private Transform _followTarget;
     private Rigidbody _body;
     private bool _inGrab = false;
 
     void Start()
     {
         // Physics Movement
         _followTarget = followObject.transform;
         _body = GetComponent<Rigidbody>();
         _body.collisionDetectionMode = CollisionDetectionMode.Continuous;
         _body.interpolation = RigidbodyInterpolation.Interpolate;
         _body.mass = 20f;
 
         // Teleport Hands
         _body.position = _followTarget.position;
         _body.rotation = _followTarget.rotation;
     }
 
     void Update()
     {
         PhysicsMove();
     }
 
     private void PhysicsMove()
     {
         // Position
         var positionWithOffset = _followTarget.position + positionOffset;
         var distance = Vector3.Distance(positionWithOffset, transform.position);
         _body.velocity = (positionWithOffset - transform.position).normalized * (followSpeed * distance);
 
         // Rotation
         var rotationWithOffset = _followTarget.rotation * Quaternion.Euler(rotationOffset);
         var q = rotationWithOffset * Quaternion.Inverse(_body.rotation);
         q.ToAngleAxis(out float angle, out Vector3 axis);
         _body.angularVelocity = axis * (angle * Mathf.Deg2Rad * rotateSpeed);
 
         // Teleport back when too far
         if(distance > teleportDistance)
         {
             _body.position = _followTarget.position;
             _body.rotation = _followTarget.rotation;
         }
     }
 }

The script is attached to a hand model with a rigidbody, and the follow-target is the XR Controller. Everything seems to be working, except the speed of the rotation.

Rarely worked with quaternions, so it might just be some basic issue.