Font Size in TextMesh with Orthographic Camera

I'm creating a 2D game. It runs in a fixed resolution of 800x600, so I set the main camera's orthographic size to 300 and set all my sprites to "1" pixels to units. Everything works as expected so far except fonts.

I laid the screen out in Photoshop using a font called Wendy. Here is what I'd like things to look like:

Nice and clean. In my Wendy font import settings, I set the font size to 20, the same font size that I use in Photoshop. In Unity, I keep the scale of (1,1,1), character size to default 1, and font size to 0 (to stick with the default import size), and the font is so tiny that I can't see it. Why!?

How does that size relationship work? If I import at size 20, my desired size, and my world is setup 1 unit to 1 pixel, why doesn't the font come in at the right size?

In searching around, I found a post that recommended that I set characterSize = desiredWorldFontSize*10/fontSize. So, 20*10/20 = 10. I put 10 in CharacterSize and that gives this which is close but still wrong:

I really would love to fully understand how font sizes work in Unity. That formula make things close, but I have no understanding of what is going on there.