No collision between objects connected by fixed joints

Hi! I'm creating a game where you build a wall of physics driven wooden beams to defend against waves of enemies (catapults). You get to place the beams during a "build phase", where physics is disabled. If you place one beam on another, they connect to each other with fixed joints. The way i've done this is that when you place a beam, it checks for collision with another beam. If there is a collision, it adds a fixed joint component and sets the other beam as the connected rigidbody.

My problem is, if you build a box, where beams that are not connected with rigidbodies meet, they collide and the whole thing becomes very unstable. What I wish is for each beam to not have collision with any of it's "parents" connected beams. I'm not using a parent / child system though, because the beams should be physics driven. I want to use fixed joints. If a structure snaps in two, these two parts should then have collision.

I'm utterly stuck. Could someone please point me in the right direction? Thanks!