VR Camera rotating/orbiting far away from its origin point

I am using Google Cardboard SDK. I have attached my camera to a car object and all of the sudden when I build the project, the camera gets pushed away from its point of origin and orbits around the car when I look around instead of just rotating around its origin.

As soon as I remove the camera as a child of the car object, everything is normal and I can look around just fine.

I have done no code, I am just using the default settings that come with Cardboard SDK (Unless I accidentally changed something, because it was working fine before).

Illustration of my problem: http://imgur.com/a/GaGmi

The camera starts inside of the car, and also stays inside of the car when running in Unity, but changes when running in VR.

Any ideas? Thanks!

EDIT: It turns out I'm only getting this issue when connecting my camera as a child with one specific 3D model.. Still very confused.