Preventing objects from shoved

Right now I allow my player (this is a First Person view game) to pick up objects with the left mouse button. When an object is picked up, it is parented to the Main camera (Camera.main.transform) and rigidbody.isKinematic is set to false on the object that is picked up (so it doesn't collide with objects in the environment). While this solves the "collide with objects in environment" problem, if the player walks to a wall (or really, any object in the room), the object will go "inside it". If the player lets go, it will be pushed outside of the room (essentially "pushed into the wall") and lost forever (given that this is a puzzle game, and this is a core puzzle piece, this is NOT a good thing... hah). At least with objects IN the room, the rigid body will push itself out of it if placed inside, so I'm not too terribly worried about that.