Unity collision detection only for raycasts

Hello again. I'm interested in how Unity's collision detection methods work.

The topic states my main question: Is there a way to disable colliders from checking collision against other colliders and instead check collisions only with raycasts.

Why would someone want that, you ask? I have a lot of objects in the scene with colliders and only need them to check against my raycasts. I reckon the performance could improve a lot with disabling the unnecessary checks.

I know that needless optimization is not time best invested in the middle of a project, but this is a question that's been stuck in my head for a while and I see no harm in getting it out of there.

While searching for an answer I bumped in another questions that woke my academical curiosity:

One would think that checking "IsTrigger" on the colliders would improve performance due to being ignored by the physics engine. Instead enabling it for some 30 objects seems to drop the FPS noticeably (not severely though). What's the explanation?

Thank you.