Smoothly move continuous moving object

Currently I am working on clone of Ping Pong type game. I want to move my ball continuously on screen. At present I have code for ball movement but it looks jittery.

 public class ContinuousMoveSinglePlayer : MonoBehaviour
 {
     private bool isStartMoving;
     private Vector3 direction;
     private float speed = 10f;
     private int opponentPaddleCounter;
 
     // Use this for initialization
     void Start ()
     {
         InitializeValues ();
     }
 
     private void InitializeValues ()
     {
         float xDirection = Random.Range (0, 2) * 2 - 1;
         float yDirection = Random.Range (0, 2) * 2 - 1;
         direction = new Vector3 (xDirection, yDirection).normalized;
     }
 
     // Update is called once per frame
     void FixedUpdate ()
     {
         if (isStartMoving) {
             rigidbody2D.velocity = direction * speed;
         }
     }
     
     void OnCollisionEnter2D (Collision2D otherCollision)
     {
         if (otherCollision.gameObject.CompareTag (Constants.TAG_OPPONENT_PADDLE)) {
             opponentPaddleCounter++;
             if (opponentPaddleCounter > Constants.BALL_SPEED_LIMIT) {
                 opponentPaddleCounter = 0;
                 speed += 1f;
             }
         }
 
         Vector3 normal = otherCollision.contacts [0].normal;
         direction = Vector3.Reflect (direction, normal);
     }
 }

I want to move ball smoothly on screen. I already tried with following code

 transform.position = Vector3.Lerp(transform.position,(transform.position + direction), Time.deltaTime * speed);

But above line creates poor collision response. It detect collision with some time delay so that it looks likes ball go inside the wall. I need your suggestions to solve this problem.