Resizing font by scale according to screen size, (More complicated than you might think)

So I am working on a pixel art game, and we decided to use pixelated font (Monkeyisland-1990-refined) Anyways, unity insisted on anti aliasing my font, I couldn't access the filter mode by inspector, so I just made a script that changed it:

 public class SetFontFilterMode : MonoBehaviour {
 
     void Awake () {
         TextMesh textMesh = GetComponent<TextMesh> ();
         Text text = GetComponent<Text> ();
         if (textMesh != null) {
             textMesh.font.material.mainTexture.filterMode = FilterMode.Point;
         } else if (text != null) {
             text.font.material.mainTexture.filterMode = FilterMode.Point;
         }
     }
 }
 

So I resuse this script on text and textmeshes, (On different scenes) But anyways, this works fine if I set the font size to 0, and it would use the default font size built into the font. I was wondering if it was possible to make it scale appropriately without using the "Best fit" option on the text component, as that sets it to non 0 text font size, which anti aliases the font, requiring me to use really large fonts and a scaled down text to make it look smooth (It's pixel text, need a huge font to make it even look okay) I was wondering if there is a better way of doing this, all my research on google gives me the good old scale down increase font size, best fit etc. And doing that is forcing me to exceed the "memory limit" of my text texture (You can't have long words with large font, there is a limit on the amount of memory of the texture output of the UI Text, (Not the TextMesh) That doesn't work, UI scaling also doesn't scale the 0 font at all, it uses pixel size, so not even the canvas scaler set to screen size relative helps :/ work son image n stuff, but not on text. So i'm considering writing my own screen space scalar if there is no other option (Which lets me scale using transform not width and height) there by allowing me to scale that way, but I thought there gotta be a better way :/ Anyways, if I find something in the mean time ill update this thread as I figure i'm not the only one hopelessly hoping that Unity people have predicted my problems and solved them before me :'(