raycast and instantiat advice

I am about to start production on a new game where the code below is one of the main functions. I learned how to write it from a tutorial and just came back to it. I made a little tech. demo works nicely the raycast finds a the object you click with mouse and instantiates a bullet from another object.

My only worry is when I click on objects further away from the inatantiated prefab the velocity seems to be much greater than closer objects! A little insight would be very welcome before starting! Thank you in advance!

 if(Input.GetMouseButtonDown(0)){
             
             Ray rayOrigin = Camera.main.ScreenPointToRay(Input.mousePosition);
             RaycastHit hitinfo;
     
             
             if(Physics.Raycast(rayOrigin, out hitinfo, distance)){
                 
                 // Var for third person raycasting
                  // Also part of third person raycasting.
                 var direction = (hitinfo.point - actor.transform.position);
                 // var testray = new Ray(actor.transform.position, direction);
                 // Draws ray from 'actor' var to mouse click.
                 // Seems to draw multiple rays at the moment. I think from the camera.
                 Debug.DrawRay(actor.transform.position, direction);
                 
                 Debug.Log("You are casting a ray");
                 Debug.DrawLine(rayOrigin.direction, hitinfo.point);
                 
                 // Instantiate.
                 Rigidbody clone;
                  
                 clone = Instantiate(bullet, actor.transform.position, transform.rotation) as Rigidbody;
                 clone.velocity = transform.TransformDirection(direction * speed);