How to enable passing through collider if speed is high enough at collision point

Basically I want a block that behaves like a normal collider, but if the player flings themselves at the block at a high enough speed then they will pass through the block. This mechanic can best be explained through the celeste dream blocks:

celeste dream blocks

So far I tried having 2 colliders. One normal and one a trigger. When the oncollisionenter function is called, I check if the player's speed is high enough, if not I do nothing. If it is high enough then I set ignorecollision for the player and block. Then when ontriggerexit is called, I revert the ignorecollision between player and block.

So far this works but is buggy. Sometimes oncollisionenter is called right after ontriggerexit so I have to use a short timer to prevent that or else my enter logic gets called after exiting the block. Another bug is that if I'm already touching the block then I will collide with it no matter what since oncollisionenter has already been called, so I just end up bouncing off the block.

Is there any way to implement this mechanic this way? Or would I have the do some heavy bounds checking logic in my update function? If so how would I approach that? Below is my current code:

 void OnCollisionEnter2D(Collision2D collision)
     {
         
         Collider2D col = collision.collider;
         if(col.tag == "Player"){
                      
             Rigidbody2D playerBody = col.attachedRigidbody;
             Player playerScript = col.gameObject.GetComponent<Player>();
 
             if(Time.realtimeSinceStartup - triggerExitTime < 0.1) {
                 return;
             }     
 
             if(collision.relativeVelocity.magnitude < 10){
                 playerBody.velocity *= 0.01f;
                 return;
             }
             
             Physics2D.IgnoreLayerCollision(8, 9, true);
 
             playerBody.gravityScale = 0;
 
             playerScript.isAllowedToFling = false;
             playerScript.hideArrowSprite();
 
             playerScript.blockParticles.GetComponent<ParticleSystem>().Play();
 
             playerBody.velocity = (collision.relativeVelocity) * BLOCK_FRICTION;
 
         }
     }
 
     public void OnTriggerExit2D(Collider2D col)
     {
         if(col.tag == "Player"){
             triggerExitTime = Time.realtimeSinceStartup;
             
             Rigidbody2D playerBody = col.attachedRigidbody;
             Player playerScript = col.gameObject.GetComponent<Player>();
 
             playerScript.isAllowedToFling = true;
 
             playerScript.blockParticles.GetComponent<ParticleSystem>().Stop();
 
             playerBody.gravityScale = 1.0f;
 
             Physics2D.IgnoreLayerCollision(8, 9, false);
 
         }
     }