Rigidbody slowly falling through mesh collider

I have objects that are supposed to be following the player character around a level.
The level is a single, large mesh with a Mesh Collider attached to it.

The player character has a CharacterController and each of the objects that follow him has a Rigidbody (Using Gravity and not Kinematic) and a Capsule Collider attached. Each FixedUpdate(), the objects set to follow the player have their desired position calculated and are translated to that new position:

Vector3 movement = desiredPosition - transform.position;
transform.Translate(movement, Space.World);

They follow the player around fairly well but I have one strange issue with their behavior: When the player stops on a slope with one of the chaser objects also on a slope, sometimes (most of the time but not always) the chaser starts slowly moving downward—passing through any colliders (including the slope it should not be passing through). The chaser will continue to do this until the player moves a significant distance away at which time it snaps to its expected location.

I'm unsure if this is a physics problem (I've tried turning up the Physics Solver Iterations, altering Interpolation, and other things to no avail) or what. It's certainly gravity that's pulling it down but it seems to do it in a fashion that make it ignore the behavior it's Follow script applies unless the player is moving.

Does anyone have suggestion on what else to look at, change, etc? It's kind of a game-breaker as it is now.