Getting character to smoothly rotate to walk on walls/ceiling

I'm working on a game where you can change the direction that gravity pulls, so the character can walk on the walls and ceiling. The character consists of a Parent gameobject, with a child view camera. When the mouse is clicked, I cast a ray, and check the normal of any hit, and the new gravity is -normal. I'm using this code to handle the rotation (current up being the normal from the raycast hit):

     Physics.gravity = -currentUp * Standpoint.GRAVITY_STRENGTH;
                 Quaternion oldRotation = targetRotation;
                 targetRotation *=  Quaternion.FromToRotation(oldRotation * Vector3.up,currentUp);
                 
                //The below line is because sometimes the above line wasn't generating the correct target rotation
                 if(targetRotation * Vector3.up != currentUp)
                 {
                     targetRotation = Quaternion.FromToRotation(Vector3.up,currentUp);
                 }

The problem I'm having is that the camera doesn't end up facing the same relative vector, it feels like you're looking left or right sometimes rather than the direction you were facing before. I think I need to orient the look vector to the new ground plane somehow, and ideas?