Changing which axis the camera FOV/screen ratio is constrained by

So I'm working for mobile and want to support both landscape and portrait orientations.

However, the problem I'm running into is this: When the screen reorients, every element which is dependent on the screen aspect changes size.

For example, in landscape, at an FOV of 20, my character is 110 pixels tall. When the screen rotates, the camera seems to 'zoom out' and the character becomes like 80 pixels tall. Even though the screen size and the FOV are identcal, Unity is somehow basing the camera view off the horizontal width only, which has shrunk, so everything else appears to shrink.

Being able to simply set which axis the FOV is based on would be nice, or if someone has an equation that will keep my 'real' FOV consistent across aspect changes....