GPU gradually raises in usage, I uncap and then recap the FPS and its drops again

Hey guys this is my first time posting here,
Been working with unity for quite a while now for many projects, however in my most recent project, Creating a mine-craft clone(with a twist) for fun, I'm running into a weird issue with the GPU performance.

I've got a head for optimization, and that is pretty much what this project is for, seeing how efficient i can be, however I'm running into a weird 'bug' with the GPU usage which i cant seem to track down and is noticeable in even fresh projects.

I have my game capped to 120fps. Using: Application.targetFrameRate = 120; (ive tried Vsync also, has the same issue)

at the start, it uses 2-4% of my GPU in task manager, over time, it will randomly spike up to 40% and stay there. I bound a key to toggle change the target frame rate to -1 (infinite) and 120, If I double tap it, unlocking the frame rate for less than half a second, the GPU usage goes back down to 2-4% for a while before eventually spiking back up, and its very repeatable and seems to not be effected by what caps the FPS or any rendering i'm actually doing

For these tests my character was standing still, and i didn't move or rotate the camera in order to try and keep consistent results. It happens when built and inside the editor

Although this is not game breaking, and the gpu controls itself when things actually get asked of it (even when pinned, having a lot of work (millions of verts as an example) it will calm down and won't be effected. just having the gpu randomly pin itself at 40% when only 5% or so is needed is kinda frustrating me.

Thankyou in advance to any of the advice, Ask any questions you guys need from the project Zinex,

EDIT: Just tested this with a blank scene and fresh project, even with nothing in the project this is still happening on unity 2018.14f1

Video

The Line of code that happens when I press the button L: Image: