Terrain Performance on Shaded Terrain

So I have a map system in my rpg that loads terrain near the player. It loads the terrain the player is on, and 8 more around it like a ring.

I have been having performance issues with the terrain. The profiler is saying Terrain.CullAllTerrains() is 70-80% of the processing in the game.

The strange thing is, I decided to drag in one of the exact same terrain pieces in the editor before pressing play, where it is then static, and compared the memory consumption. Surprisingly, the static terrain only took 0.06 Milliseconds to process culling, and the dynamic one (exact same terrain data) took 0.76 Milliseconds to process.

Turns out, The decrease in performance is caused by this line of code:

         //Assign's a Terrain Material to the Terrain
         if (Config.USE_SHADER_MATERIAL && TerrainMaterial != null)
             LoadedFloorMap.materialTemplate = TerrainMaterial;

I added a terrain shader that supports normal maps, but it's causing crazy performance issues. Is it just the shader I'm using or is it all terrain shaders?

All help is appreciated.