Speed up scrolling 2D background ends up jerky

I have a parallax background and I want to speed up the different layers (they are Quads) as long as the user does a left/right swiping motion with his finger.

I use 2 scripts for this and right now the textures jerk around every time I try to update their scrolling speed.

BackgroundScrolling.cs:

 public static GameObject BackgroundQuad;
 private float newSpeed;

 void Start () {
 
         BackgroundQuad = GameObject.FindWithTag ("BackgroundQuad");
 
     }

 void Update () {
         newSpeed = Mathf.Lerp (0f, PlayerControls.speedBackgroundQuad, 3f);
 
 BackgroundQuad.GetComponentInChildren<Renderer>().material.mainTextureOffset = new Vector2 ((Time.time * newSpeed) % 1, 0f);
     }

PlayerControls.cs:

 /// I do the standard checks for Touch.Input detection and then just increase my public static float which gets used in BackgroundScrolling.cs' Update function
 
 speedBackgroundQuad += 0.0001f;

As long as I move my finger (to constantly speed up), the textures jerk wildly. When I lift my finger they continue to run smoothly on the new speed. This is the first time I tried to do something like this and I'm confused :(

Is there any way to save my code, or do I have to use a completely different approach to parallax backgrounds? :(