SphereCast againt Terrain collider

Hey there,

I have a terrain, and I'm SphereCasting for my hit detection, the problem is: if my sphere cast hits the terrain, or at least would hit it, even if it was stopped by another collider, the performance drops from 150 fps to 10 - 40... if I replace the terrain with a box collider, its fine nad stable.

My SphereCast is infinite, just for testing. Is SphereCasting against a terrain collider really that heavy? or am I doing something wrong?

Thanks ahead, Michael.

Terrain properties: