Glitchy collisions

So, I have my cube act as a player with a player script that controls its movements and it also has a Rigidbody2D attached to it which handles gravity and such. But the thing is that when I collide with walls horizontally my cube kinda moves in for a millisecond (or a very short amount of time) and then it goes back and to where it was and it keeps on like that. How can I fix this?