How can I avoid a framerate drop when asynchronously loading scenes

I am building a large scale VR game that needs to dynamically load different parts of the world without loading screens. I am using SceneManager.LoadSceneAsync to load levels but I still have a framerate drop.

The level is loaded async fine but activation seems to be synchronous on the main thread. Is there a way to limit the amount of time the main thread will spend activating objects?

There was Application.backgroundLoadingPriority that should solve this problem but it seems to not do anything with the new scene manager system. https://docs.unity3d.com/ScriptReference/Application-backgroundLoadingPriority.html

AsyncOperation has a priority but this needs to be set before you start the operation and I only have a reference to it from LoadSceneAsync and it is started so this also doesn't work.

Is there a way to load scenes smoothly without a framerate drop in Unity?