2D Golf ball physics, force from right side doesn't work

I am trying to strike a ball from an angle and send it flying.

The ball is a gameObject with a rigidBody2D, sprite renderer and a collider.

The dot, is a gameObject with a rigidBody2D and a sprite renderer set to rotate around the circumference of the ball to denote it's position of impact .

    dot.transform.RotateAround(ball.transform.position, Vector3.forward, ballInput.rotationInput *      ballInput.rotationSpeed * Time.deltaTime);

Hitting the ball from the left side (angle 180 - 360) works fine, but hitting it from the right side (angle 0-180) seems to make it stop dead almost immediately.

I can't seem to figure out whats wrong with what should be some basic physics. Using just AddForce() without the dot's position doesn't change the behavior either.

     else if (Input.GetKey(KeyCode.Space))
     {

         if (force < maxForce)
             force += forceIncrement;
         else
             force = maxForce;
     }
     else if (Input.GetKeyUp(KeyCode.Space))
     {
         direction = ball.transform.position - dot.transform.position;
         rigid2D.AddForceAtPosition(force * direction.normalized, dot.transform.position, ForceMode2D.Force);
         force = 0f;
     }