Jittery rigidbody when two forces are applied

Hi fellow devs,

I am working on a mobile physics-based runner game where a ball moves forward when you touch the screen. The ball also follows your finger when you swipe it left or right. I'm using ForceMode.Force so the ball can climb ramps and drop down and bounce normally.

However when I move the ball left or right I get a physical wobbling effect on the ball. This is observed on the trail as well as the camera which is attached to the ball.

My guess is that the two forces (forward and left/right) don't work well together. Any suggestions? Below is my script:

 if ((Input.GetMouseButton(0) || Input.touchCount > 0))
 {
  // APPLY RIGHT/LEFT MOVEMENT
  var screenPos = Input.mousePosition; 
  screenPos.z = MousePositionZ;
  if (_camera is null) return;
  var worldPos =_camera.ScreenToWorldPoint(screenPos);
  var ballPosition = transform;
  var ballPositionNew = ballPosition.position;
  ballPositionNew.x = worldPos.x;
  ballPosition.position = ballPositionNew;
 
  // APPLY FORWARD FORCE
  _rb.AddForce(_ballForwardForce * Time.fixedDeltaTime * Vector3.forward, ForceMode.Force);
 }