Smooth movement for Gameobjects switching position

I'm working on a game and one of the main mechanics is switching around the different colored cannons using your finger. I got this mechanic to work so when you drag cannons over each other, they switch positions. Right now it is very "poppy" and instant because it's just setting positions, so that makes sense, but one thing is I want it to be smooth when the cannons switch positions. I tried using Vector.lerp but it doesn't seem to be working the way I intended. This question is continuing from this one. Here's the code I'm using to set the position of a cannon and the attempted use of lerp.

 gameObject.transform.position = Vector3.Lerp(gameObject.transform.position, new Vector3(0.21f, -4.11f, 0.51f), lerpSpeed);