Player "pinball" effect

I have a game where the player can rotate a level(basically a cube) around them by pressing a button. but when I rotate the level a certain way the level heavily pushes the player; so much so that at times the player gets flung around so fast it ignores any colliders I set up. I'm Looking for a way to stop this from happening.

So far I have tried parenting the player to the level while it was rotating but it seems to have no effect at all. Is there a way to keep the player in place while rotating the cube?