Getting pixel font to look decent

I'm in the process of creating a pixel art game using Unity. We are using the Bavaria Regular font which was made to look good at 8px and the ThinPixel7 font which is made to look good at 20px. I imported both into Unity and used them for a while, but in tweaking the game I noticed that the text is never truly sharp. I set both fonts at Hinted Raster Rendering Mode but other then that I do not know how to improve it. I figure that the Bavaria one will only look good at 8, 16, 24px etc, but that is not the case. Same goes for the ThinPixel7 one which doesn't look sharp on either 20, 40 or 60px. I know that the FontSize in the import settings is related too, but I don't know what to set it too.

My canvas' are all at Screen Space - Overlay with 100 reference pixels per unit (if that matters). Pixel perfect is also on for all the canvas' in the entire game. I'm kind of lost in Unity on how to improve how the text looks and even getting my pixel font to look good at sizes it should be designed for. What can I do?