Changing rotation also changes position?

I am working on my animation in the animation editor, every time I change the rotation of my object, the position is being slightly changed. Why is this and how can I stop it? For now I have to go in and delete the unwanted transform changes.

Ive read about the pivot being the cause of this, but is it not possible to just change rotation even if the pivot isn't completely centered?