How can I separate tree colliders from terrain colliders?

Hi. I'm making a game that has a birdseye view perspective camera that the player can control by scrolling using the screen sides or zooming using the mouse wheel. The camera has an effect that when it is within a certain distance from the ground, it will match the changes in elevation as it scrolls.

My problem is that when I add colliders to my terrain's trees, the camera tries to adjust its height to match the height of each tree that it passes over. I cannot use tag tests to get around this as the trees will always have the same tag as the terrain. I tried to test for collider type, but I either did something wrong or that method isn't possible in this scenario.

I could duplicate my terrain and remove all trees and textures and tag this duplicate as something different and only use it for camera height adjustments, but I'm worried about the effect on performance, and the issues it brings whenever I want to manipulate terrain either in editor or in game. I could also manually add in a separate game object with all of my tree colliders, but this would have similar issues as the other method as well as being pretty time consuming.

So, my question is: What is an easy, low cost way to differentiate between terrain and the trees attached to that terrain when using a raycast from a birdseye camera?

I really don't think code is that relevant here, but just in case...

 GameObject cameraObj;
 bool canZoomIn;
 
 float cameraHeightMin = 40f;
 float cameraHeightFlow = 2f;
 float terrainHeight;
 
 void SetFinalCamHeight()
 {
     Vector2 screenCenter = new Vector2(theScreenWidth / 2, theScreenHeight / 2);
     Ray testRay = cameraObj.camera.ScreenPointToRay(screenCenter);
     RaycastHit testHit;
     
     float cameraHeightFinal;
     float terrainHeightOld = terrainHeight; // storing the value of the camera height modifier from the test done in the previous update
     
     if (Physics.Raycast(testRay, out testHit))
     {
         if (testHit.collider.tag == "Ground" || testHit.collider.tag == "Ocean")
         {
             terrainHeight = testHit.point.y;
             
             // if cam height is less than the min allowed height (adjusted for terrain), sets it to the min allowed hieght, and tells the camera that it can no longer zoom in
             if (cameraObj.transform.position.y < (terrainHeight + cameraHeightMin))
             {
                 cameraHeightFinal = terrainHeight + cameraHeightMin - cameraObj.transform.position.y;
                 cameraObj.transform.position += new Vector3(0f, cameraHeightFinal, cameraHeightFinal * -0.5f);
                 // Z value adjusted to account for fact that the camera is at a 60 angle.  i.e., camera must move 1 unit back for each 2 units up in order to maintain its center of focus
                 canZoomIn = false;
             }
             
             // if the camera is above the minimum height from the ground, but still somewhat close, it should raise and lower with the terrain for visual effect
             else if (cameraObj.transform.position.y < (terrainHeight + (cameraHeightFlow * cameraHeightMin)))
             {
                 cameraHeightFinal = terrainHeight - terrainHeightOld;
                 cameraObj.transform.position += new Vector3(0f, cameraHeightFinal, cameraHeightFinal * -0.5f);
             }
         }
     }
 }