How do I get Player(CharacterController) not be affected by any objects?

So I have my Player with a CharacterController attached and DragRigidbodyScript, which works great. I want to be able to drag objects and so on, but I don't want other objects to affect my Player. When I walk into objects (like cubes) my Player sometimes just flies up in the air and behave very weird when moving objects hit the Player. Can I make the Player not be affected by objects moving into me or is there another way to pass this problem?

Thanks!