Game starts stuttering over time

Lately I was having alot of trouble with stuttering in my game. About a month ago I switched from a CharacterController to a Rigidbody controller but I have been encountering jitter/stutter eversince. At first I thought it had something to do with Cinemachine, but as of now I don't think that's the problem. The strange thing is when running my game (both in standalone as well as in editor) it runs perfectly fine in the beginning but after some time, this can be after 30 seconds or 2 minutes the background starts stuttering and eventually it will go away as well. This happens with both v-sync on and off, but with v-sync on it happens less often and you can sometimes play for 5 minutes untill the stutter starts happening.

My Cinemachine Brain has both the Update Method aswel as the Blend Update Method set to FixedUpdate and all my movement is happening in FixedUpdate aswel. For moving and rotating the Rigidbody I'm using _rb.MovePosition and _rb,MoveRotation

In my time settings I have set the Fixed Timestep to 0,01666667 (60fps) when using 0,02 (default) I get the background stutter all the time.

I have tried profiling my standalone game, but I can't seem to find reasons for the stutter and my game generally maintains high fps (320fps with v-sync off). Below you can see a screenshot of the profiler, although I admit I'm not very experienced with deepprofiling.

I have tried working with interpolation on the rigidbody, but this doesn't change anything. Same goes for changing various settings on the Cinemchine Brain. Besides that @Gregoryl explained what to use in different situations in one of his last comments in my previous topic.

Example video showing the stutter near the end.

Here you can see a recording of my game (Standalone) and you can see the stutter appearing at the 0:45 second mark.

I have also added both my movement code and a build of my game so you can see the stutter for yourself. It can be played with both keyboard and mouse as well as with an Xbox One or Ps4 controller.

A Google Drive link to the build

My RigidbodyController script