How to assign individual data to a tiles? (Unity3D Tilemap)

I want to be able to assign both static and dynamic data to a tile within the Unity Tilemap system.

When I create a new tile, like so:

 [CreateAssetMenu(menuName = "New CustomTile")]
 public class CustomTile : Tile {
    public float WalkSpeed;
 }

I am able to create "instances" of that CustomTile in the editor, as ScriptableObjects. This works well if I want to assign static data that never changes across that tile type, such as it's WalkSpeed, but how can I assign dynamic data that can change throughout the game to each individual tile, no matter what type it is?

For an example, I need each tile to hold data about its BaseTile (the initial tile placed upon world generation), Floor and Wall. As an example, if I destroy the Wall, the tile should now be drawing the Floor texture, and if I destroy the Floor, it should now show the BaseTile. For this, I need to have some system that keeps data about how much hitpoints the Floors or the Walls on that tile have, if they do exist.

I know of a solution that uses a Dictionary<Vector3Int, TileData, where I would assign all data that I need per position, but this seems very dirty and bloated and I cannot believe that there isn't a more elegant solution for all of this. Another downside of this method is that I have to manage the data (within the dictionary) and the actual drawing of the tiles (with Tilemap.SetTile()) seperately.

For clarity, I am making a Rimworld-like game.

What's an elegant solution that is simple to maintain and fulfills my needs?