How to calculate ui image size in world space?

Hello, first time posting so let me know if I am doing this wrong.

So given a ui canvas with the following settings

Render Mode: Screen Space - Camera

UI Scale Mode: Scale With Screen Size

I can provide more of the settings if you need them but I am assuming this is all that is needed.

Now with this information I would like to calculate the width and height of a ui image object in world space.

Now to give a little background, I am making a 2d grid game and want to resize the orthographic camera to fill the whole screen with the grid. That part was not to hard to figure out. The orthographic size is just half of height of the grid in world space. But now I have a UI and I need to recalculate the orthographic size to account for the ui that is on the top and bottom of the screen. So I need a way to calculate the height of the ui in world space so I can add that to the height of my grid and then calculate the orthographic size.

Here is what I have so far:

 GameObject objectivePanel = this.uiOrientation_.FindChild("ObjectivePanel");
 GameObject powerUpPanel = this.uiOrientation_.FindChild("PowerUpPanel");
 
 RectTransform objectivePanelTransform = objectivePanel.GetComponent<RectTransform>();
 RectTransform powerUpPanelTransform = powerUpPanel.GetComponent<RectTransform>();
 
 Vector2 objectivePanelMinWorldSpace = Camera.main.ScreenToWorldPoint(objectivePanelTransform.rect.min);
 Vector2 objectivePanelMaxWorldSpace = Camera.main.ScreenToWorldPoint(objectivePanelTransform.rect.max);
 
 Vector2 powerUpPanelMinWorldSpace = Camera.main.ScreenToWorldPoint(powerUpPanelTransform.rect.min);
 Vector2 powerUpPanelMaxWorldSpace = Camera.main.ScreenToWorldPoint(powerUpPanelTransform.rect.max);
 
 float powerUpPanelHeight = powerUpPanelMaxWorldSpace.x - powerUpPanelMinWorldSpace.x;
 float objectivePanelHeight = objectivePanelMaxWorldSpace.y - objectivePanelMinWorldSpace.y;

I am assuming I am missing some sort of scale / offset that I need to apply first. When I add my calculated height to the size of the grid the ui is either overlapping or, at a different resolution, there is a huge gap between the ui and the grid.

If anyone could lend some advice, that would be most appreciated.

Thanks!