How do I compress a normal map to DTX5_NM without getting inconsistent results?

Using the Default Shader (Specular Workflow) I am having difficulty getting a consistent "neutral" normal map to actually look neutral, particularly for very glossy materials at a high angle of incidence.

I am using these floor tiles, where they have been randomly rotated by multiples of 90 degrees. I've tried neutral normal maps with both 128,128,255 and 127,127,255, both as uncompressed pngs and dtx5_nm, and the rotation of the tile significantly affects how light interacts with it.

When I have no normal map whatever, they all look the same regardless of rotation.

Is there anything I can do to get this looking right other than avoiding very glossy surfaces? Is this something a heightmap would be better for?

Compressed:

Uncompressed: