Lining up Fov's

I have a game with a rifle in it. As you'd expect I have two cameras, one for hipfire and one for looking down the scope. I'm doing this with fov so I've got one with high fov and one with low fov.

The problem is if I line up a shot from the hip, and then zoom in..the crosshair no longer lines up. It aims quite a bit above the target. (Depending on how far away my target is.)

It also happens if I just change the fov of the camera instead of switching to another.

Any help would be appreciated, cheers.