Box collider acting weird.

I have never had a problem with box colliders before( In other versions of unity). Ever since my box collider slows down my player and if I change the walking speed it will be the same. The worse thing is sometimes you will walk to the speed assinged but the player will shake like crazy... It makes it look like there is a lot of lag.

I would use other colliders but my game uses physics where you can push objects around and for some reason other types of colliders wont allow to push things arround at all. At time its worse than box colliders.

Anyway to fix this?