How to calculate the rotation of a hexagon based on its six vertices position

The solution to your problem is actually quite involved (I had to write something similar for a game I'm working on, in the context of aligning the faces of polyhedra with each other). I'll try to walk you through as best as I can, but it may take some time to understand what is going on.

From what I can tell, your question is: given the location of the vertices of an arbitrarily rotated hexagon in 3D spaceVector3[] rotatedHexagonVerts = new Vector3[] {v1, v2, v3, v4, v5, v6}, you want to be able to calculate some rotation Quaternion hexagonRotation that defines the rotation of rotatedHexagonVerts relative to some neutral position (X, Y, Z rotation all equal to zero) given by Vector3[] neutralHexagonVerts = new Vector3[] {n1, n2, n3, n4, n5, n6}

As will all thing rotational, Quaternions will be your best friend. I will not assume your hexagon is symmetrical (or even flat for that matter), so this solution will work in all cases.

 private Quaternion GetHexagonRotation(Vector3[] rotatedHexagonVerts, Vector3[] neutralHexagonVerts)
 {
     Vector3 neutralRefVector1 = neutralHexagonVerts[1] - neutralHexagonVerts[0];
     Vector3 neutralRefVector2 = neutralHexagonVerts[2] - neutralHexagonVerts[0];
     Vector3 rotatedRefVector1 = rotatedHexagonVerts[1] - rotatedHexagonVerts[0];
     Vector3 rotatedRefVector2 = rotatedHexagonVerts[2] - rotatedHexagonVerts[0];
 
     Quaternion rotation1 = Quaternion.FromToRotation(neutralRefVector1, rotatedRefVector1);
     Vector3 newNeutralRefVector2 = rotation1 * neutralRefVector2;
 
     Vector3 rotatedRefVector2Perp = rotatedRefVector2 - Vector3.Project(rotatedRefVector2, rotatedRefVector1);
     Vector3 newNeutralRefVector2Perp = newNeutralRefVector2 - Vector3.Project(newNeutralRefVector2, rotatedRefVector1);
 
     float angle = Vector3.Angle(newNeutralRefVector2Perp, newRotatedRefVector2Perp);
     Quaternion rotation2 = Quaternion.AngleAxis(angle, rotatedRefVector1);
 
    return rotation2 * rotation1;
 }

There's a lot of vector math going on, so you might need to take some time to dissect each operation. Feel free to reach out for questions.

Hey @stadlernicolas26 ,

First, read @UnityedWeStand 's answer -- it's more concise, and probably has better performance than my solution below :)

However, I still think there is some educational value in seeing a different approach to the same problem, so with that said, let's do some math!

The goal is to calculate the rotation required to align an arbitrarily rotated hexagon to some given up and forward vectors. We'll calculate 2 rotations -- first aligning with the up axis, and then aligning with the forward axis. After that, we'll combine the two rotations into one that aligns with both axes in a single motion.

We can break down the problem into several smaller steps:

0) Decide what is "WORLD UP" and what is "WORLD FORWARD" -- These are the axes that you will be aligning your hexagon to.

1) Find the plane ("LOCAL UP" and "LOCAL FORWARD" vectors) that the hexagon is aligned with using the 3-point method.

2) Use the Axis-Angle method to align the plane with the WORLD UP axis.

2.A) Find the axis of rotation to align the LOCAL UP axis with the WORLD UP axis using the Cross Product.

2.B) Find the angle of rotation to align the plane with the WORLD UP axis using the Dot Product.

2.C) Create a Quaternion describing the rotation to align the plane with the WORLD UP axis.

3) Rotate the LOCAL FORWARD axis of the plane to be aligned with the WORLD UP axis using the Quaternion created in step 2.C. This produces a WORLD-UP-ALIGNED LOCAL FORWARD axis.

4) Use the Axis-Angle method again to align the WORLD-UP-ALIGNED LOCAL FORWARD axis of the plane with the WORLD FORWARD axis.

4.A) Find the axis of rotation -- No work to do here :) The first rotation aligns the plane with the WORLD UP axis. The second rotation, to align the plane with the WORLD FORWARD axis, will be around the WORLD UP axis.

4.B) Find the angle of rotation to align the WORLD-UP-ALIGNED LOCAL FORWARD axis with the WORLD FORWARD axis using the Dot Product, also using the Cross Product to check the direction of rotation required.

4.C) Create a Quaternion describing the rotation to align the WORLD-UP-ALIGNED LOCAL FORWARD axis to the WORLD-FORWARD axis.

5) Combine the two rotations created above to create a Quaternion describing a single rotation to align the plane to both axes simultaneously.

Here's a screenshot with a demo of the visualization for several hexes with different rotations and reference up/forward vectors:

Here's a code sample that implements the algorithm above to calculate the rotation and visualize the results -- Attach it to an empty GameObject and try playing around with the parameters in the inspector. GetAxisAlignmentRotation is the key function:

 using System.Collections.Generic;
 using System.Linq;
 using UnityEngine;
 
 public class HexVisualizer : MonoBehaviour
 {
     public float outerRadius = 1f;
     // Pythagorean theorem -- equilateral triangle
     float innerRadius => outerRadius * 0.866025404f;
 
     [Header("INPUT: Initial hex Rotation (degrees)")]
     [Range(-360, 360)] public float rotX;
     [Range(-360, 360)] public float rotY;
     [Range(-360, 360)] public float rotZ;
 
     [Header("INPUT: Vectors to align hex to (not normalized)")]
     public Vector3 upVectorDenorm = Vector3.up;
     Vector3 referenceUp => upVectorDenorm.normalized;
     public Vector3 forwardVectorDenorm = Vector3.forward;
     Vector3 referenceForward => forwardVectorDenorm.normalized;
 
     [Header("OUTPUT: Rotation required to align hex with reference vectors")]
     public Vector3 calculatedAntirotation;
 
     Quaternion hexRot => Quaternion.Euler(rotX, rotY, rotZ);
 
     const float referenceVectorScale = 3.0f;
 
     Vector3[] GetCornersUntranslated()
     {
         // These are the corners for a pointy-top hex centered at the origin aligned with the XZ plane
         return new[] {
             new Vector3(0f, 0f, outerRadius),                   // Top
             new Vector3(innerRadius, 0f, 0.5f * outerRadius),   // Top-Right
             new Vector3(innerRadius, 0f, -0.5f * outerRadius),  // Bottom-Right
             new Vector3(0f, 0f, -outerRadius),                  // Bottom
             new Vector3(-innerRadius, 0f, -0.5f * outerRadius), // Bottom-Left
             new Vector3(-innerRadius, 0f, 0.5f * outerRadius)   // Bottom-Right
         };
     }
 
     Vector3[] GetCornersTranslated() { return GetCornersUntranslated().Select(x => transform.position + hexRot * x).ToArray(); }
 
     Quaternion GetAxisAlignmentRotation(Vector3 localUp, Vector3 localForward, Vector3 worldUp, Vector3 worldForward)
     {
         // Align hex with the UP vector provided by the user via the Axis-Angle method
         var upRotationAxis = Vector3.Cross(worldUp, localUp).normalized;
         var upAngle = Mathf.Acos(Vector3.Dot(worldUp, localUp));
         var alignUpRot = AxisAngleToQuaternion(upRotationAxis, upAngle);
 
         // Align hex with the FORWARD vector provided by the user
         // Find which way the hex thinks is "forward" after aligning with the UP vector...
         var upAlignedLocalForward = (Quaternion.Inverse(alignUpRot) * localForward).normalized;
         // ... and compare it to the reference forward vector
         var forwardAngle = Mathf.Acos(Vector3.Dot(worldForward, upAlignedLocalForward));
         // If the cross-product is negative, invert the rotation direction
         var flip = Vector3.Cross(worldForward, upAlignedLocalForward).y < 0f;
         if (flip) { forwardAngle = -forwardAngle; }
         var alignForwardRot = AxisAngleToQuaternion(worldUp, forwardAngle);
 
         // Multiply the two rotations to get a single rotation that aligns to both axes simultaneously
         return Quaternion.Inverse(alignForwardRot) * Quaternion.Inverse(alignUpRot);
     }
 
     public Quaternion AxisAngleToQuaternion(Vector3 axis, float radians)
     {
         var s = Mathf.Sin(radians / 2);
         var x = axis.x * s;
         var y = axis.y * s;
         var z = axis.z * s;
         var w = Mathf.Cos(radians / 2);
         return new Quaternion(x, y, z, w);
     }
 
     // A hexagon's center is directly between any two opposite points
     Vector3 CalculateCenter(Vector3[] corners) { return (corners[0] + corners[3]) / 2f; }
 
     // Find the plane the hex lies on using the 3-point method
     Vector3 CalculateNormal(Vector3[] corners)
     {
         var (a, b, c) = (corners[0], corners[1], corners[2]);
         var ab = b - a;
         var ac = c - a;
         return Vector3.Cross(ab, ac).normalized;
     }
 
     // Assumes pointy-top hexes, with the top vertex first, where "forward" is bottom -> top
     Vector3 CalculateForward(Vector3[] corners)
     {
         return (corners[0] - corners[3]).normalized;
     }
 
     void OnDrawGizmos()
     {
         // Calculate the rotation to align with the reference axes
         var corners = GetCornersTranslated();
         var hexCenter = CalculateCenter(corners);
         var hexUp = CalculateNormal(corners);
         var hexForward = CalculateForward(corners);
         var antiRotation = GetAxisAlignmentRotation(hexUp, hexForward, referenceUp, referenceForward);
 
         // Draw the world up/forward vectors that we want to align to
         DrawReferenceVectors();
 
         // Draw the original hex based on input params; also draw the up-alignment axis of rotation
         var upRotationAxis = Vector3.Cross(referenceUp, hexUp).normalized;
         DrawHex(corners, Color.red);
         DrawRotationAxis(upRotationAxis);
         DrawNormals(hexCenter, corners, hexUp, Color.green);
 
         // Draw the corrected hex after rotating to align with the reference axes
         var alignedCorners = corners.Select(x => antiRotation * (x - hexCenter) + hexCenter).ToArray();
         var alignedNormal = antiRotation * hexUp;
         DrawHex(alignedCorners, Color.magenta);
         DrawNormals(hexCenter, alignedCorners, alignedNormal, Color.cyan);
 
         // Output the calculated rotation to the inspector
         calculatedAntirotation = antiRotation.eulerAngles;
     }
 
     void DrawHex(Vector3[] corners, Color col)
     {
         Gizmos.color = col;
         int numCorners = corners.Length;
         if (numCorners == 0) { return; }
 
         Gizmos.DrawLine(transform.position, corners[0]);
         for (var i = 0; i < numCorners; i++)
         {
             var j = (i + 1) % (numCorners);
             var p1 = corners[i];
             var p2 = corners[j];
             Gizmos.DrawLine(p1, p2);
         }
     }
 
     void DrawNormals(Vector3 center, Vector3[] corners, Vector3 normal, Color col)
     {
         Gizmos.color = col;
         var points = new List<Vector3>(corners) { center };
         foreach (var point in points) { Gizmos.DrawRay(point, normal); }
     }
 
     void DrawReferenceVectors()
     {
         Gizmos.color = Color.green;
         Gizmos.DrawRay(transform.position, referenceUp * referenceVectorScale);
         Gizmos.color = Color.blue;
         Gizmos.DrawRay(transform.position, referenceForward * referenceVectorScale);
     }
 
     void DrawRotationAxis(Vector3 rotationAxis)
     {
         Gizmos.color = Color.yellow;
         Gizmos.DrawRay(transform.position, rotationAxis * referenceVectorScale);
     }
 }

Ok, so that was a lot of math! How do we use the above to solve the original problem statement: "Given the points of 2 hexes, find the rotation required to align them"?

For clarity, let's rename GetAxisAlignmentRotation and its parameters for your use-case -- we're not making any logic changes, just hopefully this helps make the function make more sense for working with aligning hexagons specifically:

OLD: Quaternion GetAxisAlignmentRotation(Vector3 localUp, Vector3 localForward, Vector3 worldUp, Vector3 worldForward)

NEW: Quaternion FindRotationToAlignHexes(Vector3 hex1Up, Vector3 hex1Forward, Vector3 hex2Up, Vector3 hex2Forward)

The function signature now indicates that if you know the up and forward vectors for both hexes, it will return the rotation needed to align them.

So the remaining problem is, how do we get the up and forward vectors for the hexes given only the points?

We have functions to find these, but the names may not be intuitive. Let's rename them to be more clear what they do for hexagons:

To find "up":

OLD: Vector3 CalculateNormal(Vector3[] corners)

NEW: Vector3 FindHexUp(Vector3[] corners)

And to find "forward":

OLD: Vector3 CalculateForward(Vector3[] corners)

NEW: Vector3 FindHexForward(Vector3[] corners)

With the functions renamed, the code to solve your problem should look something like this:

 // Given the following:
 // Vector3[] hex1Corners = [the vertices for hex1]
 // Vector3[] hex2Corners = [the vertices for hex2]
 
 // We want to find the rotation to align hex1 to hex2
 var hex1Up = FindHexUp(hex1Corners);
 var hex1Forward = FindHexForward(hex1Corners);
 var hex2Up = FindHexUp(hex2Corners);
 var hex2Forward = FindHexForward(hex2Corners);
 var alignmentRotation = GetAxisAlignmentRotation(hex1Up, hex1Forward, hex2Up, hex2Forward);

Hope this helps!