How to fix(reduce) skewing of objects when changing fieldOfView

Currently I'm using fieldOfView of perspective camera to implement zoom. This was working relatively good, before I've added new objects to the scene. I'm not sure what exactly I've changed that caused this.

This is the initial setting of the camera(left) and settings after fieldOfView and rotation has changed(right):

The parents of the camera do not change position (set to 0) and do not change scale (set to 1).

Changing the fieldOfView (zooming out) has the following effect:

The cube is in root hierarchy(no parent) and has default scale(1).