Web build size cut in half but load time is still the same?

The load times for a non-streamed Web project I was working have been unacceptable, so I decided to do a major optimization pass on all the assets used by my project.

After making sure all of my textures and audio were compressed the final Web build (*.unity3d) filesize dropped from 95MB to 41MB. I was thrilled... and then utterly disappointed when I found that the load time didn't seem to change at all. I still had my 95MB build out on the server, so I tried loading them both. The times were almost exactly the same.

Am I missing something? Does Web build filesize not necessarily affect Web build load time? Is there something else I should do?