Directional Scrolling with rotating camera

Ok this might get a touch lengthy so bear with me. I'm working on a game that is using mainly contextual touches for interaction (so I'm casting a ton of rays and checking layers and tags). The basic model I am using for the camera is to have it pan around on the X and Z axes by having the player "grab" an unactionable section of play area and drag the camera around. So to get that effect I have this method in my touchController feeding into the method on my camera script.

 if (landGrabbed && touch.phase == TouchPhase.Moved && rayAtTouch(out hit, touch.position)) {
                 Vector3 camDragDist = new Vector3(touch.deltaPosition.x, 0, touch.deltaPosition.y); 
                 cameraMan.dragCamera(camDragDist);
 }

That then contacts a method on my camera and moves it about.

 public void dragCamera (Vector3 scrollDist) {
         Vector3 normalizeScrollDist = new Vector3(scrollDist.x * Time.deltaTime * dragSpeed,
             0, scrollDist.z * Time.deltaTime * dragSpeed);
         transform.position += normalizeScrollDist;
 }

Works Awesome!!! Sort of.... As long as the camera is not rotated. Once the camera rotates in the world it's axes are all different and the movement is all messed up....

What's a good way to handle this? Do I make the camera a child of an empty gameObject? That way I can move this around in worldSpace while spinning around the camera? Is there a way in code the best handle this?