The easiest approach is to not use raycasts. Attach a collider to the object you want to drag (if not already attached). Add a script to that object with OnMouseDown(), OnMouseUp, and Update() methods. In OnMouseDown, set a flag indicating dragging is active. In the OnMouseUp, reset that flag. Finger touches will be sent through the same Mouse events. Then in Update(), if the drag flag is set, position the game object's transform to the current mouse position (you'll have to figure out your own Z value if it's in 3D).

 // Something like this (it's untested, there may be some extra work involved, but the idea is generally the following and
 // it should work for an orthographic 2D camera if Z is your fwd/back axis, 3D gets more complicated, 
 // because the Z value is not necessarily into/out of screen
 void Update() {
     if (dragging) {
         Vector3 posn = Camera.main.ScreenToWorldPoint(Input.mousePosition);
         posn.z = 0; // or whatever depth you want the game object to remain at, you could save it off ahead of time and reset it here
         transform.position = posn;
     }
 }

Obviously, if you're doing this in 3D, it's much more complicated, because you have to define a depth to place the object at. If doing that, the way I would approach it is to calculate the distance to the object in the OnMouseDown event, then when you get a returned position from ScreenToWorldPoint, you normalize that value and extend it by the distance you saved off, so the object will remain at the same distance from the player no matter where they drag it.