Does having a gameobject at the origin affect performance if the player is far from the origin?

I have a randomly generating world that instantiates and destroys gameobjects around the player/camera as they move. I also have a gameobject at the origin with a script that keeps track of my inventory. As the player moves away from the origin will performance drop? Does the engine have to keep track of it? Should I move the inventory gameobject with my player?