Base map distance performance

I created a topdown game and launched it on my Android device. I had a fps of 30 when my full view was covered by a large 2048 base map texture. But when I set the basemap to a higher value (so I could see the hd textures) my fps dropped down to 20. How is such a 10fps drop possible?

I tried to remove all my terrain textures. So my whole terrain was white. The same happened: by disabling the basemap my fps dropped to 20 even when there was no texture visible. With enabling the basemap to a 0 zero distance my fps was 30 again.

How to make my topdown terrain work smooth with normal quality textures?