Lerp rotation relative to sphere

I have a sphere that I rotate my camera around while looking at it. This works fine. I also want to be able to click on a point on the sphere and have the camera zoom in, as well as adjust rotation as it zooms.

This is the code I have so far:

 if (desiredPosition != Vector3.zero) {
             float speed = 3f; // transform.position.magnitude / 10;
                         
             transform.position = Vector3.Lerp(transform.position, desiredPosition, speed * Time.deltaTime);            
             transform.localRotation = Quaternion.Lerp(transform.localRotation, 
                                                       Quaternion.Euler(-45, transform.localRotation.y, transform.localRotation.z), 
                                                       speed * Time.deltaTime);
 
             if ((transform.position - desiredPosition).sqrMagnitude < 0.01f) {
                 transform.position = desiredPosition;
                 desiredPosition = Vector3.zero;
             }
         }

This is run in Update(). desiredPosition is a point on the sphere the player clicked on. This works decently, but the problem is that depending on where the camera is around the sphere, the final rotation is lopsided (aka: camera's transform.up is not facing away from the sphere, but towards the eastern or western hemisphere, or towards the surface if on the other side of the sphere).

What I would like is for the camera's Z rotation to always be facing roughly towards the north pole, but level to the surface of the sphere and looking slightly at the surface. Think of someone standing in Australia and facing north, but looking at a point on the ground a few feet away, or standing in Alaska and facing north, also looking at a point o the ground several feet away.

What would be a good way to go about this?