Scale a 2D object based on mousePosition on click

I'm trying to scale a GameObject based on where I initially clicked on the GameObject. I'm scaling the object larger and smaller based on, while holding the mouse down, I'm moving up or down in the Y axis.

I assume that once the object has scaled, that I will have to account for the initial mouseclick position versus the offset any scaling on the object has created...

Say for example the 2D GameObject is 100x100

On initial click, the "initial mouse position"

This is what I assume the scale "pivot" would have to be after scaling up to 200x200, the "initial mouse position" taking into account the "offset" the scale has now created

I'm creating a new "pivot" GameObject, moving it to where the mouse has been clicked, then temporarily parenting my "image" GameObject underneath it.

My question is, how do I know what the offset is and cleanly continuously update it, if this is indeed the way to proceed?

 public class Interaction : MonoBehaviour {
     
     public Camera mainCamera;
 
     // scale testing
     public float ratio;
     public GameObject pivot;
     public Transform initialParent;
     public Vector3 initialMousePos;
     
     
     // Use this for initialization
     void Start () {
         
         // speed in which we scale up and down
         //TODO: make this based on how fast the mouse is moving
         ratio = 0.005f;
         
         
         // if no camera referenced, grab the main camera
         if (!mainCamera)
             mainCamera = Camera.main;
 
         // To interact we need collider, if no collider present we add one
         if (transform.gameObject.GetComponent<MeshCollider>() == null )
             transform.gameObject.AddComponent<MeshCollider>();
             
     }
     
     void OnMouseDown() {
         
         // grab initial mouse position only on first click
         initialMousePos = Input.mousePosition;
     }
     
     void OnMouseUp() {
         
         // Only proceed if the temporary pivot object exists
         if(this.pivot) {
             // parent image back to its initial parent
             transform.parent = initialParent;
             
             // destroy temporary pivot GameObject
             Destroy(this.pivot);
         }
     }
     
     void OnMouseDrag() {
         
         // Distance mouse moved must be equal or greater than 1, before we begin scaling
         if (Vector3.Distance(initialMousePos, Input.mousePosition) >= 1) {
 
             // Only proceed if we have not yet created a pivot
             if (!pivot) {
 
                 // Create a pivot game object for scaling relative to where the mouse was clicked
                 pivot = new GameObject("scalePivot");
                 
                 // parent scalePivot under UI Objects
                 pivot.transform.parent = transform.parent; 
                 
                 // store an empty Transform's parent as UI Objects
                 initialParent = transform.parent; 
                 
                 // move pivot to mouse position
                 pivot.transform.position = initialMousePos;
                 
                 // now reparent scaleable image to scalePivot
                 transform.parent = pivot.transform; 
             }
 
             // Check if we are moving mouse up to scale up, or down to scale down
             if(initialMousePos.y < Input.mousePosition.y) {
                 
                 // scaling up
                 Vector3 scale = new Vector3(pivot.transform.localScale.x + ratio, pivot.transform.localScale.y + ratio, 1);
                 pivot.transform.localScale = scale;
     
             }
             else {
                 
                 // scaling down
                 Vector3 scale = new Vector3(pivot.transform.localScale.x - ratio, pivot.transform.localScale.y - ratio, 1);
                 pivot.transform.localScale = scale;
             }
         
         }
 
     }
 }