Making mipmaps with pixel art look better?

For my textures, I am trying to go for an art style that is Minecraft-esque, where I use a lot of tiling of 32x32 textures. To preserve the hard pixelated edges, I use point-filtering for the textures. I created a test scene to show my issue.

Unfortunately, mipmaps look really bad with the low-resolution textures, as it becomes very noticeable where they transition: https://imgur.com/a/vyeRvlT

However, when I remove mipmaps, I get a lot of shimmering and ugliness at angles and distance: https://imgur.com/a/F9Cz8UA

I resized the 32x32 textures to 256x256, and used bilinear filtering with mipmaps, which looks great: https://imgur.com/a/7M0lZi9

This workaround, however, is not ideal. I want to preserve the hard edges, but up close the textures are smoothed out due to the linear filtering. Not to mention that 256x256 textures are 64 times larger than 32x32 textures, which is a waste of memory considering the art style I'm going for.

Is there a somewhat simple way that I can fix the issue? I want the mipmaps because they prevent the pixels from shimmering, but it's just too muddy and noticeable when using point-filtering for textures.