Move in direction of touch drag

Hello everyone, In my game I want to move the player by storing the initial mousePosition on GetMouseButtonDown(0), getting new mousePositions when dragging with GetMouseButton(0), getting the direction from the initial point to the dragging points with Drag-Initial.normalize, and moving the player in that direction with transform.position += direction speed Time.deltaTime; Almost like a virtual joystick. When I click once and drag, it goes in the first direction I drag, but then does not change if I move it all around. If I continuously click and drag in different directions it will always go in the first direction I drag. Here is my code:

     public float speed;
     private Vector3 click_pos;
     private Vector3 drag_pos;
     private Vector3 dir;
     
     // Update is called once per frame
     void Update () 
     {
         Plane plane = new Plane(Vector3.up, -1.5f);
         Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
         float plane_dist;
 
         if(Input.GetMouseButtonDown(0))
         {
             if(plane.Raycast(ray, out plane_dist))
             {
                 click_pos = ray.GetPoint(plane_dist);
                 Debug.Log("click point: " + click_pos);
             }
         }
         else if(Input.GetMouseButton(0))
         {
             if(plane.Raycast(ray, out plane_dist))
             {
                 drag_pos = ray.GetPoint(plane_dist);
             }
             Debug.Log("drag point: " + drag_pos);
             dir = (drag_pos - click_pos).normalized;
             Debug.Log("Direction: " + dir);
             transform.position += dir * speed * Time.deltaTime;
         }
         else if(Input.GetMouseButtonUp(0))
         {
             Debug.Log("Mouse Up");
             click_pos = Vector3.zero;
             drag_pos = Vector3.zero;
         }
     }

Can anyone tell me what is wrong? Thanks in advance!