What's the best logic to make this weapon recoil in VR?

I would like to have recoil apply to the weapon in the real world, I am accomplishing this through GetComponent<Rigidbody>().AddForceAtPosition(Vector3.up, MuzzleEndPoint.transform.position); and I would like to be able to counter the recoil by pulling the XR controller down. I know my way around Unity and I just need some ideas on how to do this.

I currently have the weapon attached to the controller via Configurable Joint. Applying the recoil using AddForceAtPosition does simulate the recoil a bit but it's not perfect and I would like a better way if possible.

I have tried attaching a configurable joint on the weapon grip and then to the second hand and that did not appear to work. I've also tried AddForceAtPosition with the force being the direction of the second hand controller. This worked the best but it's still not what I'm looking for, very jittery and unreliable.