Dynamic Layer Switching Performance

Just need to confirm that layer switching won't impact performance.

I have interactable objects on one layer and I want to move one of them to another, render that layer to a texture and then restore that object's layer in code.

It won't go away and rebuild Physics or update anything else that'll take a long time? (I only have one layer for physics for now and it's the only one ticked in the Physics Matrix. That physics layer doesn't involve this feature anyway.)