I need help in "stretching an object like rubber" in Unity3D

Hello. I am working on a mechanic where I need to stretch a primitive object like it is made of rubber. I want it to stretch in the direction of a transform up to a certain limit as long as a button is held down. Upon releasing the button, it should go back to its original shape. (Just like when you pull one side of a cube made of rubber). Here is what I did so far:

     public void MoveVertices(Vector3 point, StretchingPoint.StretchingDirection direction, float dragDistance)
     {
         for (int i = 0; i < stretchedVertices.Length; i++)
         {
             MoveVertex(i, point, direction, dragDistance);
         }
 
         stretchedMesh.vertices = stretchedVertices;
         stretchedMesh.RecalculateNormals();
     }
 
     void MoveVertex(int i, Vector3 _point, StretchingPoint.StretchingDirection _direction, float _dragDistance)
     {
         _point = transform.InverseTransformPoint(_point);
         float distance = Vector3.Distance(stretchedVertices[i], _point);
 
         Vector3 newPosition = Vector3.MoveTowards(stretchedVertices[i], _point, _dragDistance * Time.deltaTime * (1 / distance));
 
         if (_direction == StretchingPoint.StretchingDirection.Right || _direction == StretchingPoint.StretchingDirection.Left)
         {
             stretchedVertices[i] = new Vector3(newPosition.x, stretchedVertices[i].y, stretchedVertices[i].z);
         }
         else if (_direction == StretchingPoint.StretchingDirection.Top || _direction == StretchingPoint.StretchingDirection.Bottom)
         {
             stretchedVertices[i] = new Vector3(stretchedVertices[i].x, newPosition.y, stretchedVertices[i].z);
         }
     }

"MoveVertices" is called for every frame the mouse button is held down. "point" is the transform.position of the stretching point (which are placed on four sides of the cube).

You can see the results in the video below. It works somewhat fine when the object's rotation is zero. But when I rotate it, it stretches in weird directions.

https://imgur.com/a/5VlIXoE