Child colliders affecting parent rigidbody characteristics?

I have an object that consists of a container (that has a rigidbody component) and a child "arm" that can move relative to the container. This image shows the setup:

I need to have colliders for both the body and arm so that both can react to collisions from terrain etc. when the body is moving at speed (i.e. I need the colliders to not just be triggers).

I'm finding some very odd behaviour when I try to apply torque to the container's rigidbody with this setup - applying a purely vertical torque (rotating about the world Y axis) is causing the container to roll about its longitudinal axis (world z axis in the above image).

This behaviour only happens when the "end colliders" in the above image are enabled, and I'm at a loss to figure out why these child colliders are affecting the parent rigidbody so much, and what I can do to fix it.

I've been careful to ensure that all of the collider objects are not scaled or rotated (only their parent objects are rotated & positioned), I've set the centre of mass and inertia tensors for the rigidbody, and made everything symmetric, so I cannot understand what might be causing the issue.

Does anyone have any idea for what could be going wrong, or how I could achieve a similar affect without the problems (i.e. have a body object and a child arm object that can both move and collide with the world).

Thanks

EDIT: I've been doing more debugging, and it'd seem that the issue is due to the rigidbody's inertiaTensor. If I only set the inertiaTensorRotation (to Quaternion.identity) and leave the inertiaTensor to automatically calculate, the weird rotation issues don't happen. This isn't a solution however as I need to set both the inertiaTensor and the inertiaTensorRotation (to make the body behave as if it were smaller in mass/volume).