Rigidody.velocity = Direction * speed; How to get direction?

So I have a navigation script, I can't use the nav mesh and nav mesh agent due to technical reasons. But that's ok. Anyway I have a list of vector3s as waypoints that my sphere needs to pass through. I want it to roll so I need to use the rigidbody and physics engine. Anyway my code works, mostly, the only problem is that my sphere accelerates and decelerates when moving from waypoint to waypoint. I don't want this I want the speed to remain constant. My scene just has the ball and a plane.Here's my code I've attached to my sphere:

 using System.Collections.Generic;
 
 public class Nav : MonoBehaviour {
 
     [HideInInspector]
     public bool isNavigating;
     public List<Vector3> points;
 
     Rigidbody playerRB;
     int currentPoint, lastPoint;
     float speed = 5f;
 
     bool isMoving = false;
 
     // Use this for initialization
     void Start ()
     {
         // set in instaniate script
         points = new List<Vector3>()
         {
             new Vector3(5, 0, 1),
             new Vector3(5, 0, 5),
             new Vector3(8, 0, 5),
             new Vector3(8, 0, 8),
             new Vector3(14, 0, 8),
             new Vector3(14, 0, 14),
             new Vector3(20, 0, 14)
         };
 
         isNavigating = true;
 
         // leave here
         currentPoint = 0;
         lastPoint = points.Count;
         playerRB = GetComponent<Rigidbody>();
         
     }
     
     void FixedUpdate ()
     {
         if (!isNavigating) return;
 
         if (new Vector3(Mathf.Round(transform.position.x), Mathf.Round(transform.position.y), Mathf.Round(transform.position.z))  == points[currentPoint])
         {
             if (!(currentPoint + 1 > points.Count - 1))
             {
                 currentPoint++;
             }
             else
             {
                 isNavigating = false;
                 playerRB.velocity = Vector3.zero;
                 return;
             }
         }
 
         // Navigate
         Vector3 direction = (points[currentPoint] - transform.position).normalized;
         playerRB.velocity = direction * speed;
 
         
     }
 }