Why does the raycast hit only objects/faces of a certain rotation?

I am getting some very strange behaviour from a simple raycast script that I created. I have a simple FPC with the below script attached:

 using UnityEngine;
 using System.Collections;
     
 public class RaycastTest : MonoBehaviour {
 
     private Transform characterTransform;
     private Rigidbody characterRigidbody;
     public MouseLook FPCMouseLook;
     public Controller characterController;
     public Transform camera;
     public SmoothFollow cameraSmoothFollow;
     public float coverAdjustSpeed = 1;
     public float coverOffset = 0.5f;
     public float raycastLength = 2;
     public float raycastYOffset = 0.5f;
 
     // Use this for initialization
     void Start () {
 
         characterTransform = transform;
         characterRigidbody = GetComponent<Rigidbody>();;
         FPCMouseLook = GetComponentInChildren<MouseLook>();
         characterController = GetComponentInChildren<Controller>();
         cameraSmoothFollow = camera.GetComponent("SmoothFollow") as SmoothFollow;
     }
     
     // Update is called once per frame
     void Update () {
 
         RaycastHit hit;
         Vector3 raycastPosition = new Vector3(characterTransform.position.x, characterTransform.position.y + raycastYOffset, characterTransform.position.z);
 
         if (Physics.Raycast(raycastPosition, Vector3.forward, out hit, raycastLength)) {
             Vector3 rayHitPosition = hit.point;
             float distanceFromWall = Vector3.Distance(characterTransform.position, hit.point);
 
             Vector3 debugForward = transform.TransformDirection(Vector3.forward) * raycastLength;
             Debug.DrawRay(raycastPosition, debugForward, Color.yellow);
 
             Debug.Log ("Distance from Wall is: " + distanceFromWall);
         }
     }
 }

For some reason when I get within range of the left box and bottom box the raycast hits, but it does not hit for the right box or the top box not matter how close I get. Even stranger, if I start to rotate the right box while the player's raycast is with range, after rotating the box to a certain angle the raycast suddenly detects the box.

I have been trying to work out why this is happening for the last two days but I cannot understand it. All 4 boxes are exact duplicates with the blue sections being triggers and the red sections containing box colliders. Both blue and red sections are children of an empty gameobject and the FPC is a simple rigidbody with capsule collider setup. Can someone please explain to me why this is happening?