Layout Element Issue

Alright, after a failed attempt to find a similar issue on here, I have returned with another query.

I am creating a memory match game, I wanted it to be very flexible (able to choose how many cards before building the grid and then building the appropriate grid), however some of the finer elements to keep it looking nice are failing me.

So here is how I am formatting my Grid:

I first have Canvas.

Then on the Canvas and PreRuntime I have a Panel that takes up the entire canvas.

At Runtime I Insantiate (in a for loop) a panel for the first row.

In another for loop, I Instantiate the Buttons that are my Cards.

This runs until, at the moment anyway, a 4X4 grid of cards is produced.

The panels and buttons are all Prefabs

The Main Panel has a Vertical Layout Group, ideally to keep all of the panels the same size, that part works. My panels are always equally sized.

The Prefab Panels are outfitted with a Horizontal Layout Group, ideally to do the same thing with the Buttons.

Each Prefab Button is outfitted with their own Layout Element, the Preferred Width set to .25.

At Runtime I set all of the images to Preserve Aspect, with them enable they strech across the entire cell, however they are still at random widths vice the Preferred Width setting of .25.

However, This only works for the Buttons if I have all the same size sprites on each of the buttons. I am trying to have the buttons have animal images of varying size and my thoughts were that with a preferred width would set each cell on the Panel to 25% and then scale the image to fit in the cell. this does not work at all, my smaller images have a thinner cell, and my larger images have a wider cell.

If I set all of their Preferred Widths to 1250 they all look proper on the aspect ratio I've chosen. I can easily go in and script them to be screen width /4 but i shouldnt have to do it that way, in my opinion.