Player jumping from wall sometimes jump even when not collided with other wall

Hi, I'm currently making a simple game where player taps one button and dashes from one wall to another at high speed.

The problem is that when i tune the speed up to desired one(pretty fast, like 40-50) it sometimes jumps again when exiting collider but not hitting another one thus returning to initial state, kinda like jumping in the air, is there a fix to it?

I use continuous collision detection mode, set my timestamp to 0.01 and here is the code:

 public class PlayerMovement : MonoBehaviour
 {
     [SerializeField] private float speed;
     private Rigidbody2D body;
     bool direction;
     bool grounded;
 
 
     private void Awake() 
     {
         body = GetComponent<Rigidbody2D>();
     }
 
     void OnCollisionEnter2D(Collision2D col)
     {
         grounded = true;
         Debug.Log("enter");
     }
 
     void OnCollisionExit2D(Collision2D col)
     {
         grounded = false;
         Debug.Log("exit");
     }
 
     // Update is called once per frame
     private void Update()
     { 
             if(Input.GetMouseButton(0) && grounded)
                 body.velocity = new Vector2(direction ? -40 : 40, body.velocity.y);
                 direction = !direction;
             }
 }