Generate mesh from raycast positions, independent of rotations

I have simple script that uses 5 raycasts to find 5 vertices for mesh I want to make.

After that, i duplicate those vertices, and move them inwards, to create two sets, one of the larger vertex set and one of the "shrunk" set.

The end result will be a "frame" built from the smaller vertices and the larger ones.

This works fine if my raycast source is facing a certain direction. When it changes rotation, the "offsets" I use are not correct anymore as the directions I offset them by to make the smaller set of vertices is not "towards the middle" of the shape.

How can I make the "frame mesh" independent of the rotation the raycasts are in?

I have attached example images showing the vertices and the "frame" I want to make from these vertices.