Hold object at click position

I have some objects that I can pick up with the mouse or by touch, but the picking position is always at the center of the picked object. How could I pick up an object at the position where I clicked with the mouse and still let the physics do it's job, because if I edit the pivot of the object, it seems to pick up the object from a corner, but box or whatever I pick up still sits in the air in the same position as before picking it up.

Here is a snippet of my code:

     Vector3 worldPoint = Camera.main.ScreenToWorldPoint(Input.mousePosition);
     hit = Physics2D.Raycast(worldPoint, Vector2.zero, 50.0f);
      
     if (holdingObject)
     {
         pickedObject.rigidbody2D.velocity = (worldPoint - pickedObject.transform.position).normalized * 200.0f; ;
     }