Prevent a specific RB to influence another, but still collide with everything

I'm currently working on a heavily physics-based 3D game written in C#, including a custom rigidbody character controller. It is able to pick up other RB and drag them using RB.AddForce(). Now I'm stuck with a glitch as the player can be pushed by the grabbed GO, and this way is able to fly.

I wish to let the grabbed GO not to influence the player, but still collide with it and the rest of the environment and also influence other RBs. I got some solutions that work half-around, like disable the collision between these two RB, but this will cause glitches. Secondly I could set the grabbed RB kinematic, but this way it would glitch through static objects, right?

Please tell me a solution for this problem if you know:) I would appreciate the answers to be in C#, but probably I could convert it... Thanks in advance!