Maintain aspect ratio of UI element and all it's children and fit screen using black bars if necessary.

So I have a canvas with a black 10:16 image on it. The canvas scaler's UI Scale Mode is set to Scale With Screen Size. I added an Aspect Ratio Fitter component to the image and it works fine, scaling with the resolution and maintaining aspect ratio (not stretching) to always fit in any resolution (using black bars on the sides or top and bottom if necessary). It never cuts off the edge of the screen.

When I child UI elements to the image (which never changes its aspect ratio), the elements act inconsistently. They'll stay properly proportioned to their parent image if it's limited horizontally (black bars on top and bottom), but when it's limited vertically (black bars on the left and right) the proportions get all messed up and change relative to the parent causing them to overlap and go off the image and off screen.

I tried changing the Canvas Scaler's Match to 0.5 but it made no difference. All it did was change the elements' initial sizes which still weren't kept consistent after being limited vertically by the screen aspect ratio.

I tried adding Aspect Ratio Fitter to all the children elements as well, and it DOES scale properly with the screen size, but it forces the rect transform to stretch to fill the parent transform with no option to change it.

EDIT: The UI elements' rect transform settings don't show any change, despite them scaling with the viewport's height while height-limited.

Here it's being limited horizontally and properly scaling to fit and maintaining ratio and proportions of everything:

And here it's being limited vertically. Notice the black image scales and maintains ratio properly, but the child elements do not.