Moving object a certain distance at certain speed

So what i want to do is to move my object at certain speed, but make the speed depend on distance.

Because now what i have is i use Vector3.Lerp and that means it will move from point A to point B at certain speed. And i have the speed multiplied by time.deltatime. This looks fine if obejct is like 10m from target but what if its lets say 500m from target, it will move much quicker than if it was reaching target that is 10m away from it, cause it has to reach target at certain time.

So how would i do it, that it goes the same speed towards target, no matter how far the target is?

Thanks.