Click and Drag rotation similar to Scene View Gizmos

I have got a system set up to use Gizmos similar to the scene for Translation and Scale but I can get my rotation one working correctly.

ATM it works as you think at the start, you click on a line and drag around that axis to rotate it, but I am using "GetAxis" for the direction to drag in to rotate. This means when you rotate it upside down (for example) the XAxis is for the object in now also upside down which means when you drag left it now rotates right and so on.

Any help is appreciated and if you need more info then just ask (I would like to keep the script private if possible for now).