Help with combining Vectors

I'm writing a gun script and have two different things that I want to do at the same time in it. I have two parts of the script currently:

1. Sway

2. Aiming

The sway basically just adjusts a vector that's slightly behind the mouse movement which then Lerps to catch up. The aim has a preset vector that the gun Lerps to. The problem is that the two interfere with one another. When I aim, the gun won't actually reach the vector because the sway is pulling it back down. I know this is caused by the script lerping the sway back to the default, un-aimed position but I can't figure out how to structure it to just apply it to wherever the gun is if that makes sense.

Right now my script just turns off sway whenever I aim in, which is alright but it looks bad.

     //Gun Sway Variables
     private Vector3 firstPos;
     private Vector3 swaySecondPos;
     public float swayAmount = 0.05f;
     public float swayTime = 3f;
 
     //Aim Down Sights Variables
     private Vector3 aimSecondPos;
     public float aimTime = 0.05f;
     public bool aiming = false;
 
     private Vector3 curPos;
 
 
     void Start () 
     {
         //Set Start Position
         firstPos = this.transform.localPosition;
     }
 
     void Update () 
     {
         //Aim Down Sights Position
         aimSecondPos = new Vector3 (firstPos.x - 0.5f,firstPos.y + 0.1f, firstPos.z);
 
         //Gun Sway Execution
         float mouseX = Input.GetAxis ("Mouse X") * swayAmount;
         float mouseY = Input.GetAxis ("Mouse Y") * swayAmount;
         swaySecondPos = new Vector3 (firstPos.x - mouseX, firstPos.y - mouseY, firstPos.z);
 
         //Aim Down Sights Execution
         if (Input.GetMouseButton (1)) {
             aiming = true;
         } else {
             aiming = false;
         }
 
         if (aiming == true) {
             curPos = Vector3.Lerp (curPos, aimSecondPos, aimTime * Time.deltaTime);
         } else {
             curPos = Vector3.Lerp (curPos, swaySecondPos, swayTime * Time.deltaTime);
         }
 
         this.transform.localPosition = curPos;
 

Can anyone think of a way to modify this so the two work together? Thanks!

I'll update if I figure something out.