Packing Height Map with Normal

I was hoping to pack my height map value with something else such as the normal or the roughness etc...

However, it appears that the UV offset the height provides must be applied to both the albedo (and other) UVs as well as the normal. So given I had to read the normal already to get the height, I don't necessarily wont to have to read it again, that feels like it defeats the object.

Here I have the normal UVs offset by the height map using the standard parallax mapping node (same results with hand-written equivalent).

And without the height offset

This latter seems to have a ridge it and doesn't seem to look correct vs the first. So is it not possible to pack the height in with something else? Must I add the offset for the normal say?