Lots of Fixed Joints

Hi everyone, I'm currently making a 3D building demolition game. So for this game I've made a building out of a bunch of flattened cubes for floors and walls and cylinders for columns. I have them all connected together with fixed joints but the building sways. I also have them all lined up perfectly because I create the building in code. When I run it the building gets slightly shorter as it settles and I've had to make the joints stronger so they don't break from this initial force. Does anyone have any suggestions for making the joints work better, and to have this act more like a solid structure?