How can I exclude some trees from some wind zones?

I've added small spherical windzones to some characters to create an effect of rustling some foliage as they move through it. (The foliage is implemented as instances of a unity tree.) The problem is that larger trees get affected by them as well. I'd like larger trees to only be affected by big things such as explosions. A thick trunk tree shouldn't get jostled by a character moving by.

I tried using physics layers to create interaction exclusions, but wind/tree interactions don't follow the physics interaction matrix.

I also tried turning down the character windzones, turning down the tree response at every level, and turning up the wind response in the foliage to compensate. If the character windzone is strong enough to get any noticeable response at all from the foliage, then the trees still twitch.

Any ideas?