Problem with changing the direction of an Infinite scrolling background.

So, I have been trying to "wrap" my head around this, however been unsuccessful so far. I am working on an endless runner type game, which is why I need a scrolling background, but the issue lies in the fact that the player can traverse in all directions, basically, every single direction on the xy plane can be achieved and this meant that I need the background to scroll in the same direction, so when I change direction with the player the scrolling effect should do the same. My code:

speed is simply a float I use for controlling the scrolling speed

     rb = GameObject.Find("Player").GetComponent<Rigidbody>();

     Vector2 offset = new Vector2((rb.velocity.x / GameObject.Find("Player").GetComponent<PlayerMaster_V2>().speed) * Time.time * speed,(rb.velocity.y / GameObject.Find("Player").GetComponent<PlayerMaster_V2>().speed) * Time.time * speed);

     GetComponent<Renderer>().material.mainTextureOffset = offset;

Im using a repeatable texture on a quad made infinite by this script.

which does what It needs to do; it takes the players direction and feeds it to the renderer which will scroll the image in the right direction. Now the problem, though:

the moment (frame) I change the velocity of the player the entire background teleports, changes location. depending on how much the vector changed, but only for the frame I made the change, afterwards it functions fine and scrolls in the correct direction.

e.g.

Point B on the background has a velocity of (0,5) then when I change the velocity to (5,3) while it is located at point C, it will indeed start scrolling with velocity (5,3) but it will have relocated to Point A first. "Teleported"

This is ofc not what I want but I don't know what is causing it. Is it something build into the material.mainTextureOffset function, or am I missing something in my code. Your input is valued.

EDIT: the location it snaps too seems related with the direction, because point B will return to point C if the orginal Vector is applied, provided not too much time has passed.