Player is bouncing when colliding with wall object

Hey Guys, I'm new to Unity and I want to make a really basic game. I've added a mouse click movement system, so wherever I click, the player will go to that point. The problem is, when an object is between mouse click and player, the player keeps bouncing. The point, where the player is going to, gets deleted after colliding with the wall. Here's a GIF-link: http://skypit.16mb.com/luJrcn.gif