need a point in the right direction of how to do voxel destruction

im making a game where the player can dig into any wall similar to teardown just without the physics, i want to be able to do it in a way where i can build the map like normal out of cubes but have the mesh be subdevided into smaller cubes when the player starts to break it. i have a prototype of this by spawning a bunch of smaller cubes in the bounds of the wall's mesh, then the player can destroy those smaller cubes as they wish. but this has big performance problems and ive also found the collisions to be very buggy with so many small colliders. im pretty sure i know the best way to do this which is keep the walls as a single mesh just like a chunk in minecraft, but i have no idea where to start on how to do this and all the minecraft clone videos spawn the mesh procedurally and im not sure if the same thing can be done for non-procedural cubes.