Need help adding a hit point feature to rigidbody bricks getting hit with a cannonball

So what I am trying to do is create a cannonball type of game where players can shoot various types of cannonballs at enemy castles. I built the castles out of a couple hundred rigidbody bricks so that when the cannonballs hit them, they react appropriately. The other thing I want to do though is make it so that if a specific brick gets hit too many times, it can crumble/disappear, making it possible to do things like knock over a tower by breaking the bricks on the lower levels and causing it to lean.

I was thinking I could just create a script that included a basic hit point variable, and set it up so that the game calculates the damage based on the force of the cannonball hitting it. I think this method would work, but it looked like I would have to manually attach that script to each of the several hundred bricks. Is there an easier/better way to do this? Any advice or tips would be appreciated.