Getting game logic object from GameObject

I have all my game logic contained within a GameManager class and I'm only using Unity as a graphics and input engine. None of my prefabs have any Unity components relating to behaviour (as that's all contained within GameManager). Background : I'm porting an old game from XNA to Unity.

I'm using raycasting within Unity to get a GameObject from the scene when ever the mouse clicks it, this works fine and I'm able to access all components of the GameObject.

 public GameObject GetRayCastGameObject()
 {
     RaycastHit hitInfo;
         
     Ray ray = mainCamera.ScreenPointToRay(Input.mousePosition);

     Physics.Raycast(ray, out hitInfo);

     if (hitInfo.transform != null)
     {
         return hitInfo.transform.gameObject;
     }

     return null;
 }

However I'm having difficulty in translating that GameObject into an object I hold programatically in a manager class.

Specifically :

I have a PlayField class that holds an array of Blocks[X, Y]. When I click on a block in the scene I want to take that reference I have to the GameObject and pass it into my PlayField class to 'damage' the block in Blocks[X, Y].

Conversely the other way round, when creating a block, I can do quite easily. I take the raycast on a mouse click, find where it's hit in the scene (Vector3) and create a new Block within the Playfield class. This then raises a domain event to instantiate a new Block from a prefab at that location.

 var woodenCrateCreatedDomainEvent = new WoodenCrateCreatedDomainEvent(this.inputManager.GetRayCastLocation(), this.PlayerPosition.rotation);
 DomainEvents.Raise<WoodenCrateCreatedDomainEvent>(woodenCrateCreatedDomainEvent);

One solution I have thought of is to havean Identity component on every Block which will just be a GUID. I could then use GameObject.GetComponent() to retrieve the GUID and pass it into a Playfield.DamageBlock(string GUID) method. However this would mean that the Playfield class would have to lookup a GUID match in its list of Blocks to find the Block object that is hit, and when you have thousands of Blocks this will be quite slow - string matching over thousands and thousands of objects isn't exactly fast.

Is there a better way / a de facto way of translating from a Unity scene GameObject into an object that I hold within the core game logic memory ?

Sudo code :

 GameManager (empty Unity component with GameManager script) initialises.
 GameManager (script C# code) constructor called.
 GameManager initialises private class PlayField.
 GameManager (script C# code) StartUp() method called by Unity.
 GameManager calls PlayField.Generate()
 PlayField creates a private array of `Blocks[X, Y]` blocks and holds these in memory.
 PlayField calls UnityEngine.Instantiate("Block", position, rotation) for each `Block` in `Blocks[X, Y]`

Now the blocks exist in the game logic (in PlayField as Blocks[X, Y]) and as a visual GameObject in Unity

Game runs... and player in Unity clicks a block :

 GameManager does ray tracing to find which Unity `GameObject` it is (as "gameObject")
 GameManager matches `GameObject` to a game logic `Block` in the PlayField `Blocks[X, Y]` array 
 // How to do previous step as `GameObject` holds no information about `Block` in game logic or visa versa
 GameManager updates game logic to perform action on `Block` e.g. reduces health
 GameManager checks to see if `Block` is dead
 GameManager removes `Block` from `Blocks[X, Y]` removing it from the game logic
 GameManager calls UnityEngine.Destroy("gameObject") removing it from the graphics and input engine (Unity)

The one I can't seem to figure out is the "GameManager matches GameObject to a Block" step as it goes across from the Unity domain to the game logic domain.