how to create an inverse-TRS matrix directly?

So I can create a TRS matrix with Matrix4x4.TRS(pos, rot, scale), then I can transform from local-to-world using matrix.MultiplyPoint3x4(local).

And, I can get the world-to-local matrix by using matrix.inverse, and do the opposite transformation.

My question is, is there a way to create that inverse-TRS matrix directly? I'm not sure what .inverse does, but I'll like to gain as much perf as possible so this seems like an unnecessary step.

I've tried Matrix4x4.TRS(-pos, Quaternion.Inverse(rot), scale), but it doesn't work... and incidentally, I'd also like to avoid Quaternion.Inverse().