Racquet in VR, doesnt hit the ball, and sometimes it "sticks" to the ball.

Hi! Im trying to create a simple racquet in VR, basically the idea is to attach a racquet to the controller(s) and then be able to hit a ball with the racquet.

However.. Im having a few problems, one of them beeing that if I hit the ball to quick (with a high velocity) the hit wont get registered by Unitys physics engine and then the racquet simply goes through the ball.

Another issue is that if I hit the ball fast but not that fast.. it sometimes feels like the ball sort of "sticks" to the racquet before it bounces away.. and I guess thats also sort of related to the nature of the physics-engine..

Any ideas? Would it be possible to somehow calculate the collision on Update()?

Br, Inx