Change direction of ball based on where it hits paddle -- 3d breakout game

Howdy,

So I have been making a 3D breakout game. The paddle moves along the x-axis and travels "forward" in the z-axis, y remaining at 0. I have the following code for the ball's collision with the paddle:

 void OnCollisionEnter(Collision collision){
         if (collision.gameObject.tag == "Player") {
         float x = hitFactor (transform.position, collision.transform.position,collision.collider.bounds.size.x);

             Vector3 dir = new Vector3(x,0, 1).normalized;
             //Set velocity with dir * 70f;
             GetComponent<Rigidbody>().velocity = dir * 70f;
         }
 }

 float hitFactor(Vector3 ballPos, Vector3 paddlePos, float paddleWidth){
     return(ballPos.x - paddlePos.x) / paddleWidth;
 }

For the most part it works as wanted although there is one minuet problem. Let's say that the ball is traveling from left to right, if it hits the paddle on the left side, it initially travels right for the first frame and then switches towards the left (as planned). The same thing happens if the ball is coming from the right and hits the right side of the paddle.

How would I correct this from occurring? Advice would be preferred over straight code solutions if possible, but the latter would suffice. I'm just not sure where this would be corrected at.

Thanks in advance!