How to make custom handles take gameobject position offset into account?

Ok so ive made a little gif of what I have currently and it should better explain what I have currently and what I'd like to have: http://i.imgur.com/DU6g9qX.gifv

• Basically I have an array of 4 points(well currently theres two one representing the handles the other the position in combination with the gameobject but that is just the current implementation).

• I have a custom handle for each of the 4 points.

• I want the handles to stay at the blue points when I move the game object the script is attached to, but also be movable themselves. I cant figure out how to do both, just one or the other or with the discrepancy.

I cant just add the positions as the handles will fly away at any minor changes, but I am struggling to figure out how to approach this. Code below

         private int m_Corners = 4;
         public Vector3[] m_LocalCornerVertices;
         public Vector3[] m_WorldCornerVertices;
 
         public void Start()
         {
             //Initialise the arrays
             m_LocalCornerVertices = new Vector3[(1) * (m_Corners)];
             m_WorldCornerVertices = new Vector3[(1) * (m_Corners)];
 
             //Make a square to start with
             m_LocalCornerVertices[0] = new Vector3(0, 0, 0);
             m_LocalCornerVertices[1] = new Vector3(0, 0, 1);
             m_LocalCornerVertices[2] = new Vector3(1, 0, 0);
             m_LocalCornerVertices[3] = new Vector3(1, 0, 1);
         }
 
         public void Update()
         {
             for (int i = 0; i < Corners; i++)
             {
                 var oldPos = m_LocalCornerVertices[i];
                 var newPos = oldPos + transform.position;
                 m_WorldCornerVertices[i] = newPos;
             }
         }

Lastly theres a loop with floor.m_LocalCornerVertices[i] = Handles.PositionHandle(floor.m_LocalCornerVertices[i], Quaternion.identity); in an inspector script in OnSceneGui