Drag Object Via Axis Handles

I'm trying to make a control node object for use in my scene that behaves like the handles you would use in a 3D program.

That is, if I drag the X handle I can move that object on its X axis, the Y on its Y axis, and the Z on its Z axis. A picture ought to make it pretty clear:

I have code to drag an object on the plane parallel to the camera, and I imagine its a similar process but I'm having issues with the code.

             Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
             if( Physics.Raycast(ray, out hit, 500) ) {
                 if( hit.transform.tag == "draggable" ) {
                     hitObj = hit.transform.gameObject;
                     scanPos = hit.transform.position;
                     if(Input.GetMouseButtonDown(0)) {
                         screenPoint = Camera.main.WorldToScreenPoint(scanPos);
                         offset = scanPos - Camera.main.ScreenToWorldPoint(
                         new Vector3(Input.mousePosition.x, Input.mousePosition.y, screenPoint.z));    
                         drag = true;
                     }     
                 }
             }

This is the code I use to drag in 3-space. But now I want to limit to just a plane by dragging a specific handle.

I've tried doing things by having the offset only take mouseposition on the specified axis but this is chopping up the movement entirely.