Ball bounces when moving with high speed on a surface

I am creating an "Endless Runner" game where the main character is a ball. I am moving the ball with this:

 void Start()
 {
     rigidbody = GetComponent<Rigidbody> ();
     xPosition = rigidbody.position.x;
     lane = 2;
 }
 
 
 void Update()
 {
     
     Vector3 pos = rigidbody.position;
     if (Mathf.Abs (xPosition - pos.x) > 0.1f)
         pos.x = Mathf.MoveTowards (pos.x, xPosition, Mathf.Abs (xPosition - pos.x) * Time.deltaTime * drift);
     else
         rigidbody.velocity = new Vector3 (0.0f, 0.0f, rigidbody.velocity.z);
     rigidbody.position = pos;
     
     
     if (Input.GetButtonDown("Left"))
     {
         if(lane == 2 || lane == 3)
             lane--;
     }
     if (Input.GetButtonDown("Right"))
     {
         if(lane == 1 || lane == 2)
             lane++;
     }
 
     switch (lane) {
     case 1:
         xPosition = xLeft;
         break;
     case 2:
         xPosition = xCenter;
         break;
     case 3:
         xPosition = xRight;
         break;
     }
 }
 
 void FixedUpdate() {
     if (speed < 200f)
         speed += 0.1f;
     rigidbody.velocity = new Vector3 (rigidbody.velocity.x, 0f, 10f * speed * Time.deltaTime);
 }

The ball (sphere collider) is rolling on top of a plane (mesh collider), which is a part of a tunnel-like object. The plane is perfectly flat. I am using 0-bounciness materials for both the ball and the plane.

Now, when the ball reaches high speeds it starts to jiggle a bit (left/right). When I disable my y-axis nullifier then the ball starts to bounce every now and then. I want it to be stable even at high speeds and I have no idea what to do. Please help me with this.