How do I make a script for camera recoil recovery?

I already have a script which implements recoil by adding to a recoil value, which then offsets the cameras rotation. This works fine, however, when you try to do recoil control by moving the mouse down, when the recoil goes down the whole camera points at the spot you moved your mouse to. I've tried to offset this by getting how far the mouse has moved and subtracting that from the recoil amounts but it still doesn't seem to be right. I've looked online for a bit and have found practically nothing (Bad keywords maybe?) about this issue. How can I solve this?