Physics.Raycast ignore Physics Layer Matrix?

GameObject A on layer X fires (via Raycast to get hit) at GameObject B on layer Y which also has a big ol sphere collider around it on layer Z.

On the Physics Matrix I setup layer X to ignore layer Z, but the Raycast is returning the big sphere on layer Z, not the gameObject's small collider on layer Y.

I found the stuff for Raycast about IgnoreLayer http://answers.unity3d.com/questions/126285/how-to-create-a-user-layer-that-ignore-raycast.html but wasn't sure if that was needed/written before Layers were introduced and I'm just doing something wrong.

Edit 2 - I must be doing something wrong I setup a public LayerMask layermask on this script, selected the layer and hit Run, zapped bad guy and the Raycast is still returning the hit of the big outer sphere on the layer I just said to ignore. There are parent/child relationships involved here, although each parent/child is on the right layer.

Edit - in retrospect, the article talks about user layers so it must have been written after that stuff was released, I'll assume it's just a requirement and that Physics.Raycast does ignore the Physics Layer Matrix.