Make Camera smoothly orbit around my object with mouse/touch input but with limit on z-axis

I am making kind of a showroom where when I drag with mouse/touch, I want my camera to smoothly rotate (kind of orbit) around my main game object. What I have done so far is, that I have my camera parented by an empty game object (with required rotation) and script with following code is attached to the parent object of my camera.

 public float smoothness;
 private float xRot, yRot;
 const string MOUSE_X = "Mouse X", MOUSE_Y = "Mouse Y";
 float mult = 3;
 void Update()
 {
     if (Input.GetMouseButton(0))
     {
         xRot += Input.GetAxis(MOUSE_X) * mult;
         yRot -= Input.GetAxis(MOUSE_Y) * mult;
     }
     xRot = Mathf.Lerp(xRot, 0, smoothness);
     yRot = Mathf.Lerp(yRot, 0, smoothness);
     transform.eulerAngles += new Vector3(0, xRot, yRot);
 }

My problem is that I can't find a way to restrict rotation on the z-axis. I have seen allot of posts on different forums but still didn't get. I actually have tried the following code as well but it is not smooth(kind of a little continued rotation after input is removed with lerp)

 Quaternion minRot = Quaternion.Euler(0, 0, -10f);
 Quaternion maxRot = Quaternion.Euler(0, 0, 35f);
 float mult = 30;
 void Update()
 {
     if (Input.GetMouseButton(0))
     {
         xRot += Input.GetAxis(MOUSE_X) * mult;
         yRot -= Input.GetAxis(MOUSE_Y) * mult;
     }
     transform.rotation = Quaternion.Euler(0, transform.rotation.y + xRot, Mathf.Clamp(transform.rotation.z + yRot, -5, 30));
 }

When I try using Quaternion.Lerp then it sometimes kind of stuck (I think it's the gimble lock issue?) and it also starts rotating on the x-axis (which isn't required).

 transform.rotation = Quaternion.Lerp(transform.rotation,
    Quaternion.Euler(0, transform.rotation.y + xRot, Mathf.Clamp(transform.rotation.z + yRot, -5, 30)), 0.5f);

I know, I am not well versed with Quaternion and I have read that the individual euler angle axis shouldn't be modified but how can I have a smooth rotation? Any help will be appreciated.