How can I make rigidbodies react cleanly to the character collider?

What happens is this: I have a rigidbody box that gets kicked around the level. When the box is sitting on the ground, my character controller can't move it (which is to be expected, and is exactly what I'm looking for) but if the rigidbody is still in the air, my character controller pushes it around. This includes pushing it across the level just millimeters above the ground and pinning it against a wall until the CC passes through and the box goes flying off in the opposite direction. Obviously, to have a box that is immovable when sitting on the ground, yet like a beach ball when in the air, is less than desirable.

I understand why it's doing this. It makes sense, it's just not desirable. Ideally, I'd like the CC to have little to no effect on the box. Is a rigidbody CC a solution? I'd rather avoid them if possible. Any ideas?