Camera that both orbits and follows

I am attempting to make a camera which both follows a player when he moves and orbits the player on key press (think Super Mario 64). I can easily make a camera that follows a player target by setting a fixed offset value and then positioning the camera relative to the player on every update:

 void MoveCamera()
 {
     transform.position = target.position + offset;
 }

I can also easily orbit the player by using transform.rotateAround():

 void Update()
 {
     if (Input.GetKey(KeyCode.LeftArrow))
     {
         MoveHorizontal(true);
     }else if (Input.GetKey(KeyCode.RightArrow))
     {
         MoveHorizontal(false);
     }
 }

 public void MoveHorizontal(bool left)
 {
     float dir = 1;
     if (!left)
     {
         dir *= -1;
     }
     transform.RotateAround(target.position, Vector3.up, horizMove*dir);

     offset = transform.position - target.position;
 }

I cannot do both at the same time, however, because if I use a fixed offset I will break my orbit by always positioning the camera in the same place relative to the player. I have tried to resolve this by changing my offset value each time the camera moves:

 offset = transform.position - target.position;

But for some reason, this doesn't work. Every time I move the camera, it gets farther and farther away from the target, until finally it's hardly looking at him at all.

I'm thinking that there must be some say to set a specific distance from my target without setting a specific position. But I don't know how to go about it. I know I can get the magnitude of my offset, but then what do I do with it?

A few other non-working solutions I've tried:

1. connecting my camera to my player with a hinge joint. That kinda works, but it also causes some unexpected effects (like the player rotating around with the camera, the camera and player are jittery and jerky, don't follow controls properly etc.) I'm guessing that's not a viable solution, but maybe I just have the settings wrong?

2. Creating a "camera axis" object directly above my character that moves along with the character and having the camera rotate around that with a 2D distance joint. That behaves essentially just like my original solution.

So how do I do it?

Any solutions to this?