I want to use NavMesh AI AND I want to place and destroy blocks.

I've baked a navmesh and I have AI that follows the player. I want to be able to place blocks and have the enemy navigate it's way around the placed block. I also want to be able to destroy the block. You can place the block and have a navigation obstacle with "carving" ticked on, which will put a hole in the navigation mesh, but when you remove the object, the hole in the navigation mesh remains, making it so the AI avoids the space that it SHOULD be traveling through.

Having carving not ticked on will make it so the AI just gets stuck on the obstacle, trying to get through. Is there any way around this?