Scene Takes Longer to Close the Longer it's Open

I'm having a problem where, if I keep my game open for a long time (even as much as 30 seconds), trying to close it (both from playback mode and simply Alt-F4ing a built .exe) takes around 15 seconds, and this time goes up the longer the game stays open. It closes in less than 1 second if I close the game immediately after opening it. On the other hand, if I leave it open for more than about a minute, trying to close it results in ".exe is not responding" from Windows after about 30 seconds.

Despite this, my FPS and CPU performance stays constant during the entire process as shown by the stats box in the game view.

Has anyone ever experienced this? What can I do to fix it/what can I check in general to see if something is backing up?