Neighboring grid slots on a cube sphere?

So I have a cube sphere... I've been working hard to create a height terrain on it without screwing up the seams. Anyways I wrote this code to take neighboring slots and find an average height between them to smooth out the terrain. The problem is that near the seams the smoothing looks stretched, almost like its getting some neighbors twice.

  public void Smooth() {
             for (int tt = 0; tt <cube.g.g.Count; tt++) { //loops through 6 sides of cube sphere
                 for (int i = 0; i < cube.g.g[tt].g.Count; i++) { //loops through x of 2d grid
                     for (int j = 0; j < cube.g.g[tt].g[i].g.Count; j++) { //loops through y of 2d grid
                        float avg = GetAverage(tt, i, j);
                     }
                 }               
             }
         }
     
         float GetAverage(int tt, int i, int j) {
             float average_height = 0;
             float average_count = 0;
             for (int xx = (int)(average_amount*-1f); xx <= average_amount; xx++) {
                 for (int yy = (int)(average_amount*-1f); yy <= average_amount; yy++) {
     
                     Vector2 point = new Vector2(-1, -1);
                     Vector3 p = cube.g.g[tt].g[i].g[j].point;
                     p = Quaternion.AngleAxis(xx, Vector3.right) * p; //<<what I think is 
                     p = Quaternion.AngleAxis(yy, Vector3.forward) * p; //<<causing the problem
                     int ty = cube.g.getSlotfromNorm(p, cube.gridSize,ref point); //gets slot from position
                     float amount = 150;
                     if (point.x != -1 && point.y != -1) {
                         amount = Mathf.Abs(cube.g.g[ty].g[(int)point.x].g[(int)point.y].point.magnitude) - cube.radius;
                     } else {
                         print("Fail");
                     }
                       
                     average_height += amount;
                     average_count++;
                 }
             }
             average_height /= average_count;
     
             int vslot = cube.g.g[tt].g[i].g[j].vertSlot;
             cube.chunks[cube.g.g[tt].g[i].g[j].chunkSlot].vert[vslot] = cube.g.g[tt].g[i].g[j].point.normalized*(cube.radius + average_height);
             return (average_height);
         }
 

I believe using Quaternion.AngleAxis() to get the neighboring position's is causing this but I'm not entirely sure. Going around the grid array setup for the sphere (like raycasting) is not ideal because I will use similar code for editing the sphere. Anyways any advice is welcome!