Grabbed object won't stay still

Happy easter, dear Unitians.

I am making a game where one of the main elements on interaction is grabbing on moving stuff around. I got the to point where I can grab objects fine but only way I found out to keep the objects still (i.e. not floating around uncontrollably) is to make them kinematic. However, this allows them to clip trough walls which is something I don't want.

So, is there any other way to keep Rigidbodies steady on one spot while grabbing them? Here's the part of the script that controls that:

 Vector3 ObjectPos;
     Quaternion ObjectRot;
     bool canPick = true;
     bool picking = false;
     bool guipick = false;
     GameObject pickObj;
     GameObject pickref;   
 
     void Start () {
     
         pickref = GameObject.FindWithTag ("Pickupref");
         pickObj = pickref;
 
     }
 
     public void Update () {
 
         ObjectPos = transform.position;
         ObjectRot = transform.rotation;
 
         if (Input.GetMouseButtonDown (0) && canPick) {
         
             picking = true;
 
             Ray ray = camera.ViewportPointToRay(new Vector3(0.5F, 0.5F, 0));
             RaycastHit hit;
 
             if (Physics.Raycast (ray, out hit, 3) && hit.collider.gameObject.tag == "GenericObject") {
 
                 pickObj = hit.collider.gameObject;
                 hit.rigidbody.useGravity = false;
                 hit.rigidbody.isKinematic = true;
                 //hit.collider.isTrigger = true;
                 hit.transform.parent = gameObject.transform;
                 transform.position = ObjectPos;
                 transform.rotation = ObjectRot;
             }