Framerate drops from 60 to 20 because of meshrenderer?

My game's framerate occasionally drops from 60fps to around 20fps, and according to the profiler, the main cause of this is my level's base model. The entire level's base is only 4,656 polys and 2,214 vertices, and even if I turn mesh compression on high and disable shadows for it, the framerate drops still occur. It's shader size is also small being that it's only the standard shader with a 512x256 texture. My only reasoning for why this would be happening is because the level base is the inside of a space ship, and being that you're inside the model the whole time, the camera is always drawing/rendering it, but that still doesn't quite make sense because if I make the level base really small and keep it out of the camera view, or sometimes when my character is standing still doing nothing, the framerate will drop. The framerate drops won't occur if I disable it's mesh renderer though. Any help is appreciated, thanks :)