Moving objects in the direction the player is facing

Hello,

I am working on a game and got stuck on a problem. What i am trying to do is carry object in front of me (first person shooter) like in the image and move around with it. The mouse scroll wheel helps me bring the object closer or move it further from me. I am doing this by adding Input.GetAxis("Mouse ScrollWheel") to the tranform which I am pretty sure is not the right way to do it. I tried moving the object back and forth by adding force to it but as the object is floating infront of me (using Slerp) it shakes and the player throws the object instead of dropping it because of the force.

The object that i carry have a collider, a Rigidbody and a pickable script which helps me carry them around.

The problem i am facing is that if i don't disable the collider, the object collides with every other thing and acts weirdly, but if i disable the collider, the object can pass through the walls (which i don't want). I also tried to play around with the Rigidbody component like, isKinematic, sleep/wakeup, detectCollisions, freezeRotation but everything gives me one effect and take the other.

Can anyone tell me how i can achieve this effect i want to carry and move around with objects (one at a time) and bring them closer or further from the player with the help of mouse wheel, without it passing through the walls and without it rotating around weirdly on colliding with other colliders??