Very slow performance when variable has serialize tag

When i make a new layer in my tilemap editor i have a function that initialises that layer. One of the things it does it making a new byte array to hold all the tileid's in. Ofcourse i want to serialize this so i dont lose it when i enter playmode or something like that. However when i add the serialize tag making a new byte array is suddenly extremely slow even for something like 10000-20000 length it takes several seconds. The slowness seems to scale with the size of the byte array.

First the TileIDData was public but during bugtesting i made this private to ensure not some weirdness is going on outside the class. This worked until i added the serializefield tag to it which makes me think serialization is the culprit here.

Is this some bug or am i doing something very wrong here?

 public void Initialize(int _mapsize_x, int _mapsize_y, int _chunksize, int _tileresolution){
         //Debug.Log ("Initialize");
         TileResolution = _tileresolution;
         MapSize_x = _mapsize_x;
         MapSize_y = _mapsize_y;
         ChunkSize_x = _chunksize;
         ChunkSize_y = _chunksize;
         Chunks_x = (_mapsize_x / ChunkSize_x);
         Chunks_y = (_mapsize_y / ChunkSize_y);
 
         if(Chunks_x*ChunkSize_x < _mapsize_x)
             Chunks_x = Chunks_x + 1;
         if(Chunks_y*ChunkSize_y < _mapsize_y)
             Chunks_y = Chunks_y + 1;
 
         Chunks = new GameObject[Chunks_x*Chunks_y];
         ChunkData = new Chunk[(Chunks_x*Chunks_y)];
         TileIDData = new byte[MapSize_x*MapSize_y];
     }