Wheel Collider not consistant when spinning

I have a lawn mower that I am trying to control for my game. It has 4 wheel colliders on it with no suspension at all. I have perfected the feel and control of the lawn mower for my gameplay by applying force to the overall rigid body to move and turn and letting the wheel's friction slow down and stop the motion. However, over a random amount of time (could be a few seconds, could be several minutes), the wheel colliders change their resistance to spin and some of my lawn mowers become hard to turn while others will start spinning and never stop.

I've checked and the break torque and the motor torque on the colliders never change from zero. Also, sometimes I can go into the editor, pickup and drop the lawn mower to fix it though this sometimes just changes the resistance to something else.

I am working off of a 6 inch wheel (yes, inch with 1 unit=1 meter) and my current lawn mower weight is 700 units. I have also tried changing the overall scale of the lawn mower to 5 times larger than its normal 32 square inch base and 6 inch wheel, but this didn't seem to help.

Does anyone know what could be the cause of this inconsistency and/or know of a way to fix the problem?