Vive performance hit when adding a second non-VR display

I'm working on a Windows mixed reality application with Unity 2017.3.0f3 where I have a Vive headset main view for the user, but also two additional views on connected monitors. The main monitor displays the standard first-person view of the headset camera. The second monitor is added using Display.displays[1].Activate() and setting a second camera to Target Display="Display 2" and the Target Eye is set to "None (Main Display)"

This system all generally works, but I'm getting a performance lag in the headset (VR judder) when I activate the second display. I see the performance hit even if the second camera is not rendering, or if the scene is a simple sphere and skybox. The external screens don't show any lag, it's only in the headset. The problem appears both in editor and build. Any ideas what the source/solution might be? As a test, I move the controller back and forth in front of me while wearing the headset, and it's much smoother with only one display active versus the two. Could this be related to differences in screen refresh?