2D Bounce gains energy each time

I managed to get what I wanted (it's not perfect elastic bounce but for my project will do just fine). Thanks to @davidcox70 for the tip.
Here is my setup: I removed the physics material from the rigidbody and left it to null. All the other settings for the rigidbody remain untouched. Then I created a script and attached it to the ball. The script has:

 [RequireComponent(typeof(Rigidbody2D))]
 public class BallController : MonoBehaviour
 {
     private Vector2 preHitVelocity;
     private Rigidbody2D rb;
 
     void Start()
     {
         rb = GetComponent<Rigidbody2D>();
     }
 
     void Update()
     {
         preHitVelocity = rb.velocity;
     }
     
     void OnCollisionStay2D(Collision2D other)
     {
         Vector2 avgNormal = Vector2.zero;
         for (int i = 0; i < other.contactCount; i++)
         {
             avgNormal += other.GetContact(i).normal;
         }
         avgNormal /= other.contactCount;
         rb.velocity = Vector2.Reflect(preHitVelocity, avgNormal);
     }
 }
 

This works by caching the velocity of the ball each frame in the Update() method.
Then in the OnCollisionStay2D() method (I initially used OnCollisionEnter2D() but it didn't work for me don't know why) I calculate the average normal of the collision and then I use it with the preHitVelocity to reflect the velocity of the rigidbody.

This gives me an almost perfect bounce. Sometimes the ball will bounce slightly lower then it should, but on the next bounce it will go back up to the right height. Tested it a bunch and seems to be a stable solution. I'm not sure if it's the optimal one though...