AI - Raycast hitting itself

EDIT: After more debugging I have found that when that code block seems it actually is. I put a debug statement within the if Raycast itself but before the if debug check. Basically the raycast is so small I don't see the Gizmo being drawn. The raycast is hitting itself.

I guess the new question becomes why isn't this ignoring itself. I want to ignore the AI layer but it isn't.

How do I modify my "InOffsetRaycast" function to ignore the AI layer only but keep all others?

Original Post

I am getting some weird logic. The function labeled "CanSeeObject" will return true/false if the player is within the enemies AI sight.

1. is in angle

2. is within distance

3. Raycast can hit the player

The issue i am getting is 1 and 2 execute perfect but 3 will only execute if the player is on the AI's left side (according to my debug tests). If he goes to the right side it doesn't even try to execute step 3. Super weird.

CanSeeObject

 public bool CanSeeObject(GameObject enemy, Transform eyes, float fieldOfView, float range, bool debugging=false)
             {
                 Vector3 direction = (enemy.transform.position - eyes.position)+Vector3.up;
                 // If the angle between forward and where the player is, is less than half the angle of view...
                 if (Vector3.Angle(direction, eyes.forward) <= (fieldOfView / 2) ) // within field of view
                 {
                     if (Vector3.Distance(eyes.position, enemy.transform.position) <= range)
                     {
                         if (InOffSetRaycast(enemy.transform, eyes, 0f, debugging) ||
                             InOffSetRaycast(enemy.transform, eyes, -0.5f, debugging) ||
                             InOffSetRaycast(enemy.transform, eyes, 0.5f, debugging) ||
                             InOffSetRaycast(enemy.transform, eyes, 1f, debugging) ||
                             InOffSetRaycast(enemy.transform, eyes, -1f, debugging) ||
                             InOffSetRaycast(enemy.transform, eyes, -1.5f, debugging) ||
                             InOffSetRaycast(enemy.transform, eyes, 1.5f, debugging) ||
                             InOffSetRaycast(enemy.transform, eyes, 2.0f, debugging) ||
                             InOffSetRaycast(enemy.transform, eyes, -2.0f, debugging))
                         {
                             return true;
                         }
                     }
                 }
                 return false;
             }

InOffSetRaycast

 bool InOffSetRaycast(Transform enemy, Transform eyes, float offset, bool debugging=false)
             {
                 Vector3 direction = (enemy.position - eyes.position);
                 RaycastHit hit;
 //                int ignoreLayer = ~(1 << eyes.gameObject.layer);
                 int ignoreLayer = eyes.gameObject.layer;
                 direction = (offset == 0) ? direction : direction + (Vector3.up * offset);
                 if (Physics.Raycast(eyes.position, direction, out hit, ignoreLayer)) //draw raycast straight to object from eyes
                 {
                     if (hit.transform.root.gameObject == enemy.gameObject) //Hit the object?
                     {
                         if (debugging == true)
                         {
                             Debug.DrawRay(eyes.position, direction+(Vector3.up*offset), Color.red, 1.0f);
                             return true;
                         }
                         return true;
                     }
                 }
                 else if (debugging == true)
                 {
                     Debug.DrawRay(eyes.position, direction+(Vector3.up*offset), Color.blue, 1.0f);
                 }
                 return false;
             }

At first I thought it was a True/False logic thing... thinking false was getting the higher precedence but then I realized the gizmos weren't even being drawn. Here are some pictures to help illustrate my problem. I'm really confused here.

Is anyone able to see something the looks off about this logic?

Always hits on the left side

Will not execute "InOffSetRaycast" function if on right side