Overlaying 3D video on top of rendered world in VR

Hey everyone,

Does anyone have any suggestions on how to overlay 360 video that has been chroma-keyed on top of a 3d rendered world? I'm having trouble figuring out keep my two worlds in line. I'm thinking I would let my main camera render my 3d rendered world and have another camera that renders only my video and that I rotate my video inversely to the orientation of my VR headset. Does that make sense? Does anyone have any suggestions?

Cheers,

Paul