Rigidbody slowly floats out of collision box problem

Hey guys!

Right now I'm testing some stuff before starting creating my game. I got stuck with collide box thingy.

I created Sans character, gave him ability to teleport in game, and whenever player teleports into wall character will slowly slide from the collide box zone.

Here's video: https://www.youtube.com/watch?v=c1yCKTT-vw4&feature=youtu.be

Here's picture of my colliders:

Here's my character settings:

Here's my Update code:

 void Update()
     {
         if (Input.GetKey (KeyCode.Q)) {
             T_Q.color = new Color (0, 255, 255, 255);
 
             if (Q_Stacks > 0 && Input.GetKeyDown (KeyCode.UpArrow)) 
             {
                 Q_Stacks--;
                 T_QStacks.text = Q_Stacks.ToString ();
                 ani.Play ("TP_Player");
                 rb2d.position = new Vector2 (rb2d.position.x, rb2d.position.y + TPdistance);
                 ani.Play ("TPEnd_Player");
                 T_Q.color = new Color (0,255,255,255);
             } 
             else if (Q_Stacks > 0 && Input.GetKeyDown (KeyCode.DownArrow)) 
             {
                 Q_Stacks--;
                 T_QStacks.text = Q_Stacks.ToString ();
                 ani.Play ("TP_Player");
                 rb2d.position = new Vector2 (rb2d.position.x, rb2d.position.y - TPdistance);
                 ani.Play ("TPEnd_Player");
                 T_Q.color = new Color (0,255,255,255);
             } 
             else if (Q_Stacks > 0 && Input.GetKeyDown (KeyCode.LeftArrow)) 
             {
                 Q_Stacks--;
                 T_QStacks.text = Q_Stacks.ToString ();
                 ani.Play ("TP_Player");
                 rb2d.position = new Vector2 (rb2d.position.x - TPdistance, rb2d.position.y);
                 ani.Play ("TPEnd_Player");
                 T_Q.color = new Color (0,255,255,255);
             } 
             else if (Q_Stacks > 0 && Input.GetKeyDown (KeyCode.RightArrow)) 
             {
                 Q_Stacks--;
                 T_QStacks.text = Q_Stacks.ToString ();
                 ani.Play ("TP_Player");
                 rb2d.position = new Vector2 (rb2d.position.x + TPdistance, rb2d.position.y);
                 ani.Play ("TPEnd_Player");
                 T_Q.color = new Color (0,255,255,255);
             }
         } 
         else 
         {
             T_Q.color = new Color (255, 255, 255, 255);
         }

It worked good before, it instantly teleported character back as soon as he would enter wall. I tried setting my code to FixedUpdate too, it makes movement and teleportation use very bad and still makes character to slowly float out of the wall!

Any ideas? :P