Cardboard VR: Drag object towards player but have it stop just before

I've been looking for questions that might ask about the problem I'm facing, but so far I haven't found anything that really gets at what I'm trying to do.

The setup is Cardboard VR, with a player using a phone and the single button. When the reticle is over an object, I want the player to be able to hit the button and pull the object towards her, but have it stop a unit or so in front of the player so it's not occupying the same world space. The speed of the move will be affected by the mass of the object's rigidbody, so the player can "feel" some weight of different objects. When it stops, I then want the object to become a child of the reticle so that weight is no longer a factor, and when you move it around it's not becoming a pain. (Although if I could get the object to follow a distance behind the reticle, that would be cool, but the reticle exists basically only on the screen and I haven't figured out how to use screenPosition for that yet.)

Here is what I have at the moment, after going through several different permutations:

A cube, to which Dragger.cs (since I originally planned on "dragging" these objects") is attached. I have a click event for Pointer Down linked to the AttachObject() function, and Pointer Up linked to DetachObject(). The result of the below code is that the cube approaches the player, but then passes right through and continues flying off the edge of the map. It's a bit of a mess as I've been trying different things. Anyone have any suggestions? I am stumped, and only now realize I probably should have asked for help a week ago.

 using System.Collections;
 using System.Collections.Generic;
 using UnityEngine;
 
 public class Dragger : MonoBehaviour {
 
     
     public bool lineBreaks = true;
 
     Camera cam;
 
     private GameObject ReticleObject;
     private Transform Reticle;
     private bool IsHeld;
 
     private Vector3 dist;
     private Rigidbody rigidBody;
     private float speed;
     private Vector3 offsetPoint;
 
     // Use this for initialization
     void Start () {
         IsHeld = false;
         cam = Camera.main;
         ReticleObject = GameObject.Find("GvrReticlePointer");
         Reticle = ReticleObject.GetComponent<Transform>();
         offsetPoint = cam.transform.position + (transform.forward * 2); 
 
 
         rigidBody = GetComponent<Rigidbody>();
         // Let's make sure we have a rigidbody for the weight to be simulated
         if (!rigidBody)
         {
             rigidBody = this.gameObject.AddComponent<Rigidbody>();
         }
         speed = 5.0f / rigidBody.mass;
         dist = transform.position - offsetPoint;
     }
 
     // Update is called once per frame
     void Update () {
         if(IsHeld)
         {
             rigidBody.useGravity = false;
             Vector3 direction = offsetPoint - transform.position;
 // One thing I was trying
 //  transform.position = Vector3.MoveTowards(transform.position, offsetPoint, speed * Time.deltaTime);
             rigidBody.AddForce(speed * direction);
             if (transform.position == offsetPoint)
             {
                 rigidBody.velocity = Vector3.zero;
                 rigidBody.angularVelocity = Vector3.zero;
                 transform.SetParent(Reticle);
             }
             
         }
         else
         {
             rigidBody.useGravity = true;
             this.transform.SetParent(null);
         }
     } 
 
     public void AttachObject()
     {
         IsHeld = true;
         
     }
 
     public void DetachObject()
     {
         IsHeld = false;
     }
 }