How to modify a movement step by step to continuous movement?

So I'm trying to update my current script from a movement step by step (defined by units) to a continuous movement, but I've been having some troubles with it Right now, it changes direction whenever you press any of the arrow keys and also rotates to it's desired direction, like in the following script:

         public float speed = 0.4f;
         Vector2 dest = Vector2.zero;
     
         private int direction = 0;
     
         void Start () {
             dest = transform.position;
         }
         
         void Update () {
     
             Vector2 p = Vector2.MoveTowards(transform.position, dest, speed);
             GetComponent<Rigidbody2D>().MovePosition(p);
     
             if ((Vector2)transform.position == dest) {
     
                 if (Input.GetKey(KeyCode.UpArrow) && valid(Vector2.up)) {
                     direction = 1;
                     transform.eulerAngles = new Vector3(0, 0, 0); 
                     dest = (Vector2)transform.position + new Vector2(0, 2);
                 }
                 if (Input.GetKey(KeyCode.DownArrow) && valid(Vector2.down)) {
                     direction = 3;
                     transform.eulerAngles = new Vector3(0, 0, 180);
                     dest = (Vector2)transform.position + new Vector2(0, -2);
                 }
                 if (Input.GetKey(KeyCode.LeftArrow) && valid(Vector2.left)) {
                     direction = 4;
                     transform.eulerAngles = new Vector3(0, 0, 90);
                     dest = (Vector2)transform.position + new Vector2(-2, 0);
                 }
                 if (Input.GetKey(KeyCode.RightArrow) && valid(Vector2.right)) {
                     direction = 2;
                     transform.eulerAngles = new Vector3(0, 0, -90);
                     dest = (Vector2)transform.position + new Vector2(2, 0);
                 }
             }
 

And I have the following script to detect the colliders:

 bool valid(Vector2 dir) {
         Vector2 pos = transform.position;
         RaycastHit2D hit = Physics2D.Linecast(pos + dir, pos);
         return (hit.collider == GetComponent<Collider2D>());
         //return true;
     }

And the Result is something like this: https://i.gyazo.com/8182c66af84794146515d837c0427521.gif

I tried to turn into a continous movement, but the only way I made it work + - is the following:

  void Update() {
 
         Vector2 p = Vector2.MoveTowards(transform.position, dest, speed);
         GetComponent<Rigidbody2D>().MovePosition(p);
 
             if (Input.GetKey(KeyCode.UpArrow) && valid(Vector2.up)) {
                 direction = 1;
                 transform.eulerAngles = new Vector3(0, 0, 0);
             }
             if (Input.GetKey(KeyCode.DownArrow) && valid(Vector2.down)) {
                 direction = 3;
                 transform.eulerAngles = new Vector3(0, 0, 180);
             }
             if (Input.GetKey(KeyCode.LeftArrow) && valid(Vector2.left)) {
                 direction = 4;
                 transform.eulerAngles = new Vector3(0, 0, 90);
             }
             if (Input.GetKey(KeyCode.RightArrow) && valid(Vector2.right)) {
                 direction = 2;
                 transform.eulerAngles = new Vector3(0, 0, -90);
             }
 
         
         // up
         if (direction == 1) {
             dest = (Vector2)transform.position + new Vector2(0, 2);
         }
 
         // right
         if (direction == 2) {
             dest = (Vector2)transform.position + new Vector2(2, 0);
         }
 
         // down
         if (direction == 3) {
             dest = (Vector2)transform.position + new Vector2(0, -2);
         }
 
         // left
         if (direction == 4) {
             dest = (Vector2)transform.position + new Vector2(-2, 0);
         }
     }

And the result is this: https://i.gyazo.com/61f6323b64006247a58f4abd42792416.gif

The movement there is continous, but the problem there is whenever the car is between 2 colliders, it can go against each other if it changes direction (like in the beggining of that gif) and also the car does some random spins. I know it's giving that problem cause I took this line out of it:

 if ((Vector2)transform.position == dest)

But if I leave it there, whenever the car starts, it goes against the 1st collider and can't move or make a turn, it's simply stopped.

Any help to solve this problem would be nice.