Realistic ball left-right movement

I'm trying to work on getting realistic ball movement, but without using rigidbody/The built-in unity Physics because what I'm working on doesn't have "accurate physics". So I already have the movement for going forward and slowing down/speeding up. Because in this case the ball is always moving forward with a certain speed. But I need to figure out a way to make the ball turn left and right using "force". The force is applied on a perpendicular direction every time you apply it. So the way I have the rotation working now isn't physically correct. Right now I'm just adjusting the rotation of the object and then calculating the forward vector and adding it to the position. But the way I adjust the rotation is just by adding the same amount as long as the "A" or "D" button is pressed.

             if (Input.GetKey(KeyCode.A) || Input.GetKey(KeyCode.LeftArrow))
             {
                 transform.eulerAngles += Vector3.down * Speed  * Time.deltaTime;
             }
             if (Input.GetKey(KeyCode.D) || Input.GetKey(KeyCode.RightArrow))
             {
                 transform.eulerAngles += Vector3.up * Speed  * Time.deltaTime;
             }

So the way I have it now is really simplistic but not physically correct.