Object keeps moving when using Vector3.Lerp on transform.position

Hi all, I have an issue where I want a nice smooth movement when I pickup an object and move it with the joypad. It works, kind of, that the object sometimes moves towards a direction without stopping, usually it keeps moving along the Z axis and X axis without stopping and I have to counter it with the joypad controls. I can stop it, but this random wandering movement is a pain.

The thing is, I dont think it should if I have used the transform and lerp function correctly?

I first execute the pickup function, which sets the object in front of the camera, floating in the air. Then I use the Carry function to move the object by adding the getAxis input as a modifier and then lerping between the co-ordinates.

I think this is right, but I cant see the issue here.

     void Carry(GameObject o)
     {
         //Quaternion target = Quaternion.Euler(rModX, 0, rModY);
 
         modX = modX + Input.GetAxis("Horizontal") * Time.deltaTime;
         modZ = modZ + Input.GetAxis("Vertical") * Time.deltaTime;
         modY = modY + Input.GetAxis("Move up and Down") * Time.deltaTime;
 
         modifier.x = modX;
         modifier.y = modY;
         modifier.z = modZ;
 
         o.transform.position = Vector3.Lerp(o.transform.position, originPoint + modifier, Time.deltaTime * cameraSmooth);
 
     }
 
     void Pickup()
     {
         if(Input.GetButtonDown("Pickup and Drop"))
         {
             originPoint = mainCamera.transform.position + mainCamera.transform.forward * objectCarryDistance;
             Ray ray = mainCamera.camera.ScreenPointToRay(new Vector3(screenX, screenY));
             RaycastHit hit;
             if (Physics.Raycast(ray, out hit))
             {
                 Object_Script_Pickup p = hit.collider.GetComponent<Object_Script_Pickup>();
                 if (p != null)
                 {
                     isCarrying = true;
                     carriedObject = p.gameObject;
                     p.rigidbody.useGravity = false;
                 }
             }
         }
     }

Please can someone see any issues with this code?