2D camera to won't move correctly with distance

I have an annoying issue with my camera script. I am trying to have the camera move up down left right with a raw movement of my mouse.

The problem is the movement seems to be significantly slower the more negative the Z value is for the camera which is frustrating.

I have demonstrated the problem with two animated images:

Zoomed in: Z = - 10 http://i.imgur.com/1tEsmsR.gif

Zoomed out: Z = -50 http://i.imgur.com/kySFwb1.gif

I want the movement to be equal to the mouse regardless of camera distance. But am unsure how to do this.

This is my current camera scroll script, i tried to correct the issue by applying a speed variable which used the camera's Z position but it didn't work:

 private float speed;
 
     void Start(){
         speed = transform.position.z; //trying to fix speed issue 
     }
 
     void Update () {
         if (Input.GetMouseButton (1)) {
             if (Input.GetAxis ("Mouse X") > 0) {
             transform.position += new Vector3 (
                             Input.GetAxisRaw ("Mouse X") * Time.deltaTime * speed,
                             Input.GetAxisRaw ("Mouse Y") * Time.deltaTime * speed, 0.0f);
             }
 
             else if (Input.GetAxis ("Mouse X") < 0) {
             transform.position += new Vector3 (
                             Input.GetAxisRaw ("Mouse X") * Time.deltaTime * speed, 
                             Input.GetAxisRaw ("Mouse Y") * Time.deltaTime * speed, 0.0f);
             }
         }
     }

My camera is in perspective projection.

I hope you can help explain my mistake here. :)