Animation causes unwanted rigidbody motion

I have a scene where a baseball player is hitting a ball towards a target (fielder). The ball has a rigidbody with use gravity true and with a physics material to make the bouncing as realistic as possible. The hitter has an animator with two states, idle and swinging.

To make the ball move I use a lookAtRotation to aim the ball towards the fielder and then rigidbody.AddForce to move the ball.

I created a testscene where I built the real scene in steps. When there is no hitter it works fine. The ball moves to the target every time. Also when I add the hitter, but don't play the animation the ball behaves fine.

But as soon as I play the animation in the test scene, just as in the real scene with the hitter the ball moves in all kinds of random directions. I first thought it might be that the bat collides with the ball, but even if I move the ball out of reach of the bat it still behaves strange. The swinging state of the animator has a script that sets a parameter to launch the ball when it is halfway through the swing (using OnStateUpdate).

Any thoughts? Thanks.