Spawned objects randomly not destroyed

I'm having some serious problem figuring out why red and blue balls are not destroyed in the same amount when I spawn them in a close space.

Here is a simulation where i spawn 100 red and 100 blue balls. https://www.youtube.com/watch?v=0Khs9rVCBMg

Blue balls have a script like this, where they just update the destruction counter by 1 upon destruction

 public class BlueBall : MonoBehaviour
 {
     private SpawnManager spawnManager;
     private void Awake() => spawnManager = FindObjectOfType<SpawnManager>();
     private void OnDestroy() => spawnManager.AddBlueDestroyed();
 }

Red balls have a script like this, where they will destroy themselves and a blue ball upon collision

 private void OnCollisionEnter(Collision collision)
     {             
         if (collision.gameObject.CompareTag("Blue") && dying == false)
         {                     
             dying = true;
             Destroy(collision.gameObject);
             Destroy(gameObject);
             spawnManager.AddDestroyCount();
         }     
     }

I'm setting dying to true here to prevent a red ball from destroying multiple blue balls (as actual destruction happens at the end of frame not immediately). The problem i am having is that much more red balls are destroyed than blue balls, even though there is only one place of destruction and it destroys both. As blue balls are destroyed before the blue balls in the code, we know that there is no error in this code before that, which would prevent the code from getting to the point it destroys the blue balls.

This mismatch only happens when I spawn the balls close together. If I change the scale of the room I spawn then from 5x5 to 50x50 the number of balls destroyed will be equal. What is causing this?

public git repo with the project can be found at https://github.com/endasil/DestructionProblem if someone wants to look at the exact details. Would really appreciate it if someone could help me figure out the exact mechanics of what is happening here because I spent hours trying to understand how this works.