NavMesh Agent causing huge lag

I tried to find what was causing lags and eventually I found that it was navmesh agent. I did try to look into rendering objects, but all is cubes (with specular material) and a plane, the player and enemy have script. When I start the game it is fine, when reached near enemy the fps drops as low as 3 frames. I remove navmesh component at runtime, fps is back on 100+ frames.

For rendering I used occlusion culling, but it did not make any better, as I am quite sure that navmesh is causing issue.

Tris: 2.1k, Verts:5.7k, FPS: 3.1 (318ms) is what stats show. There is no OnGui() function. I am changing the navmesh destination to player when it comes closer to enemy (in enemy script), if this can cause any computation issue?.

Player and Enemy are both cubes. There are walls (also cubes) and these cubes are baked on plane for navigation. Any solutions?