Need help with Rigidbody/Gravity issue

I'm making a prototype for an infinite runner game where the main gameplay aspect is switching gravity. However, what I have now is causing something weird to happen. At certain spots in the level, the player (for now, a cube) just seems to hit a "bump" or something, and bounces up into the air. The same thing happens if the player is on the "ceiling". It happens at the same moments too, whenever the player gets up to a certain speed, perhaps?

I've tried messing with the friction coefficients on the floor/ceiling's Physic Material, and it seems to only change when the player hits this invisible snag and bounces up. Unlocking the rotation in the Rigidbody just causes the player to spin when it bounces.

This is really frustrating, I'm including a link to my project in case anyone could be able to take a look and figure it out. I'd be super grateful.

Project link