Does a RayCast detect small colliders?

I am trying to do a button, an actual world button. And the way I check if the player is close enough to press it, is through a RayCast.

I'm using Camera.main.transform.forward for my Ray direction, and the player's position for my Ray origin. Problem is, when looking right at the button (I can actually see the Ray passing through it), it doesn't "find" it.

Here's my code: void FixedUpdate () { RaycastHit hit; Ray lineOfSight = new Ray (transform.position, Camera.main.transform.forward); //Debug.Log (lineOfSight.ToString()); Debug.DrawRay (transform.position, Camera.main.transform.forward * maxDistance);

         if (Physics.Raycast (lineOfSight, out hit, maxDistance)) {
             //Debug.Log ("Hit something!");
             if (hit.collider.tag == "Button") {
                 Debug.Log ("Found the button!");
             } else {
                 Debug.Log ("If statement was false");
                 if (hit.collider.tag == "Environment") {
                     Debug.Log ("Ground");
                 }
             }
         }
     }

I am using the tags correctly. that's why the Environment tag is there, because it find the ground, but not the button.

Here's a screenshot of both the button and the Ray.

I have tried making it a LOT larger, and it seemed to be found only when close enough to the camera itself (and the collider too). Might it be a problem with the origin of the Ray?