Collision with camera inspection target

Hi guys, I have this question/problem related to collisions. So the main camera in my game is to fallow an "inspection target" that is pretty much just a transform. Now, I'm moving the inspection target with the mouse by adding appropriate Vector3 to its position so there are no rigid bodies involved yet. Now it all works as it should but I want to add another crucial feature: I want prevent the inspection target from going into the floor, walls etc. The question is what is the right way of doing it? I'm quite capable of programming it myself if someone just points me in the right direction by explaining the theory behind it. I'm assuming I will have to add some sort of collider to my inspection target and maybe a rigid body as well. Mind that I do have some physics affected rigid bodies in the scene and I don't want the inspection target to collide with them. Also all walls etc are in default layer to make things simpler.

Any help would be greatly appreciated.

Cheers,

Kamil