Fixed joint not acting as expected

I am trying to make a door is kept shut with a plank. (see pic1) The door is supposed to stay shut unless the plank is removed. However, I found a weird scenario when the door is pushed from the other side. The yellow part of the door started to deform as I apply force. (see pic 2) The yellow parts are actually parented to the door and I even added a Fixed Joint to fix them to the wall. I suppose Fixed Joints never break under any circumstances, so this was really unexpected. How could I fix them?