Is it possible to import my own .ogg files?

I am wondering if it is possible to use ogg files that I encode outside of Unity with audio software (such as Amadeus Pro for example). The reason I ask is this:

I take an 83MB .wav file and convert it to an .ogg with Amadeus using "quality setting 1" with Variable Bit Rate(VBR) rather then choosing a constant or average bit rate - which are considered inferior ways to encode audio. The result is a decent sounding file that is only 1.9MB, which is great!

BUT, when I put that into my Unity project, it defaults to converting it to a certain bit rate, with this default option:

Original/Compressed: 1.9MB/11.2MB

Even if I go all the way down to 45kbps this is the result:

Original/Compressed: 1.9/3.2MB

So, it wants to take my VBR encoded .ogg, and make it much bigger. Is there anyway to just use my .ogg as is? Is there some technical streaming issue with VBR encoded audio that maybe I am just not aware of? My game is a music game with a ton of audio, so I am trying to get this part right.

Thanks!