Lag with trees that use colliders?

Hey, so I built a awesome terrain and realized the tree's have no colliders, so I made a new prefab with capsule colliders scaled the the right size, and when I refresh the scene with more than 2 tree types (out of 8) having colliders it gets immensely laggy- under 10fps If i go back to the tree's without colliders my fps goes back up to 90+fps. Whats the issue here?

There are alot of tree's in the scene however their spread across a 2000x2000 terrain, and its not all tree's its patches. Tree distance, detail distance, etc,etc are all low on the terrain settings.