Drag 2D object conflict with object rotation

Hi, I have a 2D project with sprites as game objects and I've made them draggable when clicked on. They have a 2D collider.

The problem is that, if I set the Z rotation of the object to anything other than 0, when clicked on, the object does not follow the mouse position anymore and goes very quickly far away until error Invalid worldAABB. Object is too large or too far away from the origin pops up.

Here's the drag script :

 using System.Collections;
 using System.Collections.Generic;
 using UnityEngine;
 
 public class Drag : MonoBehaviour
 {
     bool isDragging;
 
    
 
     private void OnMouseDown()
     {
         isDragging = true;
         
     }
 
     private void OnMouseUp()
     {
         
         isDragging = false;
         
     }
 
     // Update is called once per frame
     void Update()
     {
         if(isDragging)
         {
             Vector2 mousePosition = Camera.main.ScreenToWorldPoint(Input.mousePosition)- transform.position;
             transform.Translate(mousePosition);
 
             Debug.Log("Mouse position is : " + Camera.main.ScreenToWorldPoint(Input.mousePosition)  + "Object position is : " + transform.position);
 
         }
         
         
     }
 }

Thanks

Edit : After some research, by clamping the value of the translate, it turns out that the prefab rotated is actually orbiting around the position of the cursor with it distance to the center getting bigger at each rotation

Edit 2 : Here is a short video showing the issue https://streamable.com/yyix9v