Dynamic setting of sprites cause many draw calls

I'm building a 2D game and dynamically loading the sprites from resources (so I can use a config file to specify which sprites belong to which item). I'm getting LOTS of draw calls as I instantiate more and more objects, even though they use the same 4 sprites (which I only load once with Resources.Load()).

Any ideas?

Edit: Code to reproduce -

 void Start () {
     Sprite[] sprites = new Sprite[] { 
         Resources.Load<Sprite>("Sprites/BuildingSprites/Farm"),
         Resources.Load<Sprite>("Sprites/BuildingSprites/Lab")
     };
 
     for (int x = -10; x < 10; x++) {
         for (int y = -10; y < 10; y++) {
             GameObject obj = new GameObject();
             obj.transform.position = new Vector3(x, y, 0);
             obj.AddComponent<SpriteRenderer>().sprite = sprites[Random.Range(0, 2)];
         }
     }
 }

Edit 2 - Same thing happens if I load the sprites from the inspector... So it seems like setting the sprite property of a SpriteRenderer causes it to create a new material, which will result in a separate draw call.

Any ideas on how to set the sprites dynamically without getting tons of draw calls?