How to represent a dynamic tile like map in Unity

I am not that familiar with unity and my question is quite newbish in terms of what I want to accomplish which is a game that I can represent and interact with like a tiled based system

to help understand I have an example:

I have a 3x3 block of gameobjects on top of a single 'floor' gameobject which represents a 3x3 game 'map'

the functionality I have is as follows.

if I click on the game object I wish to destroy, it is destroyed using the following code

     if (Input.GetButtonDown("Fire1"))
     {
         RaycastHit hit;
         var ray = Camera.main.ScreenPointToRay (Input.mousePosition);
         if (Physics.Raycast (ray, out hit, 100.0f))
         {
             Debug.DrawLine(Camera.main.ScreenPointToRay (Input.mousePosition).origin, hit.point);
             GameObject.Destroy( hit.collider.gameObject);
         }
     }

Here are the before and after screenshots

before

after

my question is how can I replicate this functionality using a 256x256 map where I am not using GameObjects (as I cant use 65536 game objects to represent my map). The 'wall' sections need to be collidable. I'm quite stuck with this problem and I'm not particularly sure what I should be even searching for to get answers.

I'll update the post as required as I'm not sure how to explain this better.

Thanks for taking the time

Because a picture explains a thousand words using my modest talent at Microsoft Paint I have done an artistic impression of what the tile based map 'thingy' would look like if I carved a 3x3 hole in the middle of the map with the camera looking down it.

Edit: It wont let me put more than two pictures up so my artistic talent will have to wait another day