Move a tile (sprite) with drag and drop in Unity 2D

I would like to move a tile with drag and drop in Unity 2D. The tile is a sprite. The scene is an 'Unblock me' or 'Blocked in' like gameplay.

Because the tiles in real life correspond to physical objects it seemed be to a good idea to model them with colliders and rigidbody. The border of the table surrunded with invisible colliders. I hoped these will constrain the moves of the tiles realistic, when the player moves them.

Then I implemented a simple (mouse based) drag and drop behavior which is worked perfectly except the moved tile penetrates to other tiles and the border, and sometimes jumps over them. Then I learned if I am overriding physics by explicitly setting transforms position (which I do exactly in my drag and drop implementation), this will happen. OK I accept, I should set only forces, ect. on rigidbody never directly the position.

Now the question:

I am stuck here. I still want to drag and drop like user experience, and some realistic visual result. When in the real life a player moves a tile, it seems it is "glued" to its finger. How can I achieve this (ot at least similar) with just applying forces? Any suggestions or point similar existing sample/blog code?

(I know as a backup plan I can omit all the physics and constrain the tile positions by code, and create some tweens to move the tiles. Is the real solution (what I am asking for) so complicated I should vote on this backup plan?)