Rolling ball on many boxes causes bounce

If I make a simple rolling ball (like the roll-a-ball demo) with a sphere collider, it rolls great on a big quad (like a plane).

If I substitute the plane for a bunch of boxes, it seems OK at slow speeds. However, at faster speeds, it starts to bounce more and more on the seams between the boxes.

Why should this occur? The boxes are tightly packed, so the top surface of them is effectively a plane. Yet the ball does not roll as smooth as on a plane.

How can I eliminate this bounce to get smooth rolling? I don't want to have to redo lots of level geometry to remove box colliders and change them to planes customized for the top and sides of everything, that's a nightmare of work.

Is there another workaround? Maybe make everything smaller and move at slower speed? How do others solve this problem?