Problem with scrolling list of buttons in UI

Hi, I have a UI table with list of buttons. On button press down it should slightly scale up and on press up it should scale back. Here is a code I use for that:

 public class BuyCoinsButtonScript : MonoBehaviour, IPointerDownHandler, IPointerUpHandler
 {
     public void OnButtonClick()
     {
         // Do something
     }
 
     public void OnPointerDown(PointerEventData eventData)
     {
         this.gameObject.transform.localScale = new Vector3(1.1f, 1.1f, 1.0f);
     }
 
     public void OnPointerUp(PointerEventData eventData)
     {
         this.gameObject.transform.localScale = new Vector3(1.0f, 1.0f, 1.0f);
     }
 }

The problem is that when I scroll down/up the list, the button I put my finger call OnPointerUp/OnPointerDown immediately. But I don't want it while scrolling.

How to fix the problem above?

Thank you