Mouse movement disables Vsync (leading to very bad stuttering)

To give some more details to the title: When I play my game, vsync seems to be enabled, but not working perfectly. I get around 77fps, if I add a few Debug.Logs that run on Update() i can get it down to 66fps. I'm guessing this is because my scene right now is very very empty. (A terrain and 4 static meshes, each below 1k tri)

The real problem though: If I play in editor and wiggle my mouse, for some reason my framerate will start going higher and higher, as if Vsync disengages completely. This leads to very nasty stuttering. What's also weird is that if I build my game (Win32) and try the same thing there, sometimes if I wiggle the mouse nonstop the game actually freezes. Sound will still work, but the frame on screen won't update and even my game timer (which uses InvokeRepeating) won't update. This doesn't happen very often, but I figure I should definitely mention this.

If I keep the mouse alone, stuttering only happens 1/30 of the time and is very neglectable. I tried playing a webgl build once and I had the same problem there, but it did stutter even without moving the mouse. (The WebGL build uses the fastest quality settings though, which means no vsync and probably (couldn't check) very very high framerates, which would make these stutters even more obvious)

I tried disabling Vsync and using Application.targetFrameRate instead but that didn't change anything. I checked the profiler, and I noticed that when everything works WaitForTargetFPS will use about 70% of CPU. If I wiggle the mouse and check those frames, WaitForTargetFPS uses 0.0%, probably because it doesn't actually work anymore.