Need help with spy glass to see through objects.

I have a mechanic in a new project of mine that lets you see through objects in a sense. Right now, I have the "glass" as an object in front of what I want to see through, with a camera attached to the glass with a render texture on the back of the glass. Works as it should, as I think I'm doing it right.

Problem I'm having is that when I move the spyglass, it moves what it sees, or makes it look like that. Adjusting the FOV makes the effect less noticeable, but this is not ideal because I would have to do this every time and it takes a great deal of fine tuning.

My guess is the problem is caused by the position of the parent, and the relationship between the camera and the render texture. My problem seems a bit specific, as I usually just slap it into Google to put me on the right path but I can't seem to find the right phrasing to get me some results. Any help would be appreciated.

Demonstration. http://i.imgur.com/S3TO9tP.gifv