Problem lerping between two rotations with mouselook

I'm currently trying to make a camera script that moves & rotates the camera to the target transforms rotation & position over time, whilst still 'working' with both the mouse look scripts on my character.

Currently, the positional movement works well:

 //Interpolate the cameras to the sights positon (sW is sight weight)
 myTransform.position = Vector3.Lerp(myTransform.position, adsTrans.position, sW);

But, I need to correct the rotation so that it matches the target transforms rotation:

as soon as I try and correct the rotation of the camera, so it is aligned correctly, the camera ends up aligned on the correct axis, but the rotation isn't right.

 myTransform.rotation = Quaternion.Lerp(myTransform.rotation, adsTrans.rotation, sW);

I'm almost certain that there's some simple thing I'm doing wrong, but I can't figure out exactly what it is, and I'm not an expert on quaternions. I've tried using Transform.LookAt, and messed about with relative rotations using Quaternion.Inverse, but nothing so far has worked. Any suggestions are appreciated.