windows standalone freezes when closed (X button)

I recently added a multiplayer mode to my game, but while or after playing a multiplayer match if I press the close (X) button the unity window freezes completely. I cannot even close it with task manager, I have to wait for a few minutes before it decides to close. It must be something to do with the multiplayer as it closes fine before playing multiplayer, but not while playing multiplayer or even after.

The last line of the output log does not show anything helpful: 'Log: connection {1} has been disconnected by timeout; address {::ffff:XX:XX:XX:XX:28000}' (replacing XX with my ip).

How do I fix this?