Any way to add physical box colliders to an object with Character Controller ?

I'm currently developing a multiplayer third person shooter using Unity 5.2 and UNet, where players are controlling airships. Very few things are missing to consider the game as "playable" in an alpha release.

I recently worked on a test map, so I added obstacles like trees, and try colliding with them. But here is the problem : There is no collision.

I followed a few tutorials to learn about Unity 5 and UNet, and these tutorials were using Character Controller for player movements, so I started with this and kept it that way, thinking that would be okay even if it's not a humanoid character. I reduced the capsule size to a minimal value and added a box collider to my player prefabs. I naively thought there was no problem because this box collider was properly detecting collision with player shots (using raycast), applying damage, etc. But the only problem is with physical collisions. It acts like a trigger collider, working with all non-physical collisions. I verified the colliders of both players and obstacles, none is trigger. I tried to add a rigidbody, and then it was detecting physical collisions, but badly messing with the player prefab, making it shake, changing supposedly static values. If I put the rigidbody to kinematic, the box collider is ignored again.

I read somewhere that's because Character Controller is handling physics by itself, so I tried to create an empty child object and put the rigidbody and box collider there, but same result. Also, if I try to collide using the capsule collider of the character controller, the collision is working perfectly. So, my question is : Is there a way to deal with this problem, or should I completely rework the controlling system using rigidbodies, which would be a little annoying because I made complex movements using both keyboard and mouse, and I'm now afraid to mess with it ?

Thanks in advance !