Best way to handle large, tile-based 2D worlds with arbitrarily removable tiles?

Hi, I am more or less trying to make a Terraria-like proof of concept.

That is, the player will move around in a large, rectangular (512 * 64 uniformly sized tiles) 2D world that has been completely generated before hand (that is the easy part), and will be able to remove tiles pretty much arbitrarily. I should note I am not concerned with making stranded tiles be affected by gravity.

If the world were static, it would be a simple matter of grouping rectangles of identical tiles together in a single plane, tiling the texture appropriately, and creating colliders for the resulting rectangles that are "exposed to the air".

If it's implemented this way though, for any tile that gets removed, all rectangle the tile was a part of will have to be destroyed and recalculated along with colliders to compensate for the newly created gap.

This seems pretty ineffective, and a bit dirty to be honest.

Is there an cleaner or more effective way to set up and/or remove these tiles and generate an appropriately sized collider for the resulting hole?

If it matters, I store where the tiles should go in a 2D array of ints, with 0 representing no tile and >0 representing the various types (which are simply different textures).