NewtonVR fix. Translating a point and rotation inline with changes from another gameobject.

Hi, I make NewtonVR, a free, opensource, physics based, interaction framework for vr. The meat of NewtonVR is being able to hold a vr controller over a virtual object, hold a button, and "pick up" that item. The way we currently do this is by creating a transform (PickupTransform) at the point the item is at, then setting the parent of that transform to the controller (NVRHand).
See here: https://github.com/TomorrowTodayLabs/NewtonVR/blob/master/Assets/NewtonVR/NVRInteractableItem.cs#L180

Then, each FIxedUpdate we set the velocity and angular velocity equal to (distance / time) to get the item's position to match the controller's position plus the initial offset of position and rotation. To see this in action, check out this old blog post: http://www.vrinflux.com/newton-vr-physics-based-interaction-on-the-vive/

However, we've had a variety of issues with simply distance/time which I suspect is do to the position of the transform not being updated quite inline with the position of the rigidbody.
See here: https://github.com/TomorrowTodayLabs/NewtonVR/blob/master/Assets/NewtonVR/NVRInteractableItem.cs#L88

So what I'm trying to do is translate the "create a transform" solution to a more math based approach that solely relies on Rigidbody.position and Rigidbody.rotation. But I'm having some significant issues with that. On pickup I get a bunch of variables that might be helpful:

             PickupPointItem = this.Rigidbody.position;
             PickupRotationItem = this.Rigidbody.rotation;
             PickupPointHand = hand.Rigidbody.position;
             PickupRotationHand = hand.Rigidbody.rotation;
             PickupPointDiff = PickupPointHand - PickupPointItem;
             PickupRotationDelta = Quaternion.Inverse(PickupRotationHand) * PickupRotationItem;

Here's my current thought process for getting the deltas I need to apply the velocity:
For rotation I want to get the delta from the hand's initial rotation to the hands current rotation (currentHandRotationDelta). Then I want to apply that rotation to the initial rotation of the item (PickupRotationItem) to get the target rotation. Then the delta between the current rotation and that target rotation should be correct (but doesn't seem to be). Here's my code:

                 Quaternion currentHandRotationDelta = Quaternion.Inverse(PickupRotationHand) * AttachedHand.Rigidbody.rotation;
                 Quaternion targetRotation = PickupRotationItem * currentHandRotationDelta;
                 rotationDelta = Quaternion.Inverse(this.Rigidbody.rotation) * targetRotation;

Position is more complicated. I get the currentPickupPoint, which is the position on the object that the hand should have been over when it was first picked up. I calculate that by getting the point of the current item position plus the initial position delta and rotating that around the pivot of the current item position. Then I take the difference in position from that calculated point and the current rigidbody to get the global delta. Then I use that plus the current hand position as a point to rotate around the pivot of the hand by the rotation of the delta from the initial hand rotation to the current hand rotation. Here's the relevant code for that:

                 Vector3 currentPickupPoint = RotatePointAroundPivot(PickupPointDiff + this.Rigidbody.position, this.Rigidbody.position, Quaternion.Inverse(PickupRotationItem) * this.Rigidbody.rotation);
                 Vector3 currentDiff = this.Rigidbody.position - currentPickupPoint;
 
                 Vector3 targetPosition = RotatePointAroundPivot(AttachedHand.Rigidbody.position + currentDiff, AttachedHand.Rigidbody.position, Quaternion.Inverse(PickupRotationHand) * AttachedHand.Rigidbody.rotation);
 
 
                 positionDelta = targetPosition - this.Rigidbody.position;

Here's the RotatePointAroundPivot method:

         private Vector3 RotatePointAroundPivot(Vector3 point, Vector3 pivot, Quaternion rotation)
         {
             Vector3 dir = point - pivot; // get point direction relative to pivot
             dir = rotation * dir; // rotate it
             point = dir + pivot; // calculate rotated point
             return point; // return it
         }

Here's a link to the nonworking code on the development branch of our github repo: https://github.com/TomorrowTodayLabs/NewtonVR/blob/development/Assets/NewtonVR/NVRInteractableItem.cs This doesn't seem to be working at all :D But I'm not super sure why. I have no math training past basic algebra but I imagine this is relatively straight forward at higher math levels. What am I missing?