Vector3 question: How to know right axis when using pinch punch gestures to scale object in virtual reality?

I have an object that can be scaled in virtual reality using Vive touch controllers. The issue: the pinch punch gestures for scaling relate to the world, but the object to scale is already rotated. How do I know along which axis the user is intending to scale?

Here's a sample video of how it should work, with the sample code to achieve this below: https://www.youtube.com/watch?v=bJwgtoyedcI

As it shows, only one axis (the biggest stretch) is meant to be scaled a time. However, in this video example the cube was carefully rotated for the code to work. Rotated anyway else, and the wrong scaling axis will be applied. How would I go about this? And to clarify, I would like to only change the localScale, not permanently apply the scaling to a parent transform. Thanks for any help!

 // had been trying eulerAngles as well as localEulerAngles
 Vector3 originalAngles = (Vector3)transformToHandle.eulerAngles;
 
 // trying to reset object's rotation
 transformToHandle.Rotate(-originalAngles);

 /*
 // (an alternative was to rotate the pinch punch points along their center)
 Vector3 centerToBePivot = Vector3.Lerp(startPos1, startPos2, 0.5f);
 Vector3 anglesToRotate = -Universe.lastTransformHandled.eulerAngles;
 startPos1 = RotatePointAroundPivot(startPos1, centerToBePivot, anglesToRotate);
 endPos1 = RotatePointAroundPivot(endPos1, centerToBePivot, anglesToRotate);
 startPos2 = RotatePointAroundPivot(startPos2, centerToBePivot, anglesToRotate);
 endPos2 = RotatePointAroundPivot(endPos2, centerToBePivot, anglesToRotate);
 */


 // preparing some values
 Vector3 scale = transformToHandle.localScale;
 
 Vector3 startPos1 = positionEmptyClickStarted;
 Vector3 endPos1 = transform.position;
 Vector3 startPos2 = otherDotScript.positionEmptyClickStarted;
 Vector3 endPos2 = otherDot.transform.position;
 
 float leftRightDistanceStart = Vector3.Distance(startPos1, startPos2);
 float leftRightDistanceEnd = Vector3.Distance(endPos1, endPos2);
 
 float growth = leftRightDistanceEnd - leftRightDistanceStart;
 
 // along which axis do we want to stretch? Pick biggest
 float xStretch = Mathf.Abs(startPos1.x - endPos1.x) + Mathf.Abs(startPos2.x - endPos2.x);
 float yStretch = Mathf.Abs(startPos1.y - endPos1.y) + Mathf.Abs(startPos2.y - endPos2.y);
 float zStretch = Mathf.Abs(startPos1.z - endPos1.z) + Mathf.Abs(startPos2.z - endPos2.z);
 
 // apply the scaling now
 if ( xStretch >= yStretch && xStretch >= zStretch ) {
     scale.x = scaleAtStart.x + growth;
 }
 else if ( yStretch >= xStretch && yStretch >= zStretch ) {
     scale.y = scaleAtStart.y + growth;
 }
 else if ( zStretch >= xStretch && zStretch >= yStretch ) {
     scale.z = scaleAtStart.z + growth;
 }
 
 // ensuring scaling won't go into negative or too small
 float min = 0.01f;
 if (scale.x < min) { scale.x = min; }
 if (scale.y < min) { scale.y = min; }
 if (scale.z < min) { scale.z = min; }
 
 // reapply scale and rotate back to how it was
 transformToHandle.localScale = scale;
 
 transformToHandle.eulerAngles = originalAngles;