Excluding Gameobjects/Colliders from reacting with Collision Flags?

Hello,

I recently run into a problem, I don't know how to deal with:

I'm creating a game with a lot of swordfighting. My characters are all having CharacterControllers and the detection if the sword hits a character works with Collisions (swords have BoxColliders). That's fine, except one thing:

When I swing my sword upwards from the ground and it hits an enemy from below, he "climbs" on the sword. I lift him up. It seems as if the CollisionFlags also react on the sword-GameObject.

Now my question: Is it possible to exclude a Gameobject from reacting with CollisionFlags? If yes, how to do so?

I tried with IgnoreCollision in various manners, but it didn't work :(

Would be glad if someone knew help ;-) And sorry for maybe noob question...