Unity 5 - My raycast is hitting my player, and I can't figure out why

I'm working to get a camera that doesn't clip through objects. To achieve this, I'm using the following code:

     //Here I declare that my player's position will be the raycast start, and the camera's position will be the end
     Vector3 rayStart = target.position;
     Vector3 rayEnd = transform.position;

     //The ray direction will be normalized later, since we'll be basing the ray distance off of it
     Vector3 rayDir = rayEnd - rayStart;

     //We set the ray distance, and check to make sure the distance isn't 0 (the camera would be at exactly the same position as the player, in this case)
     float rayDist = rayDir.magnitude;
     if(rayDist <= 0.0f)
     {
         return;
     }

     //Now we normalize the ray direction, by dividing it by its magnitude
     rayDir /= rayDist;

     //Here, I make the actual raycast. It goes from start to end, in the correct direction, checks for a radius of 10.0f, and ignores everything except for the player and any layer labelled as "Ignore Raycast" (at least, it should). If we didn't hit anything, we return, since there's nothing to do.
     RaycastHit[] hitInfos = Physics.SphereCastAll(rayStart, ObstacleCheckRadius, rayDir, rayDist,  m_RaycastHitMask);
     if(hitInfos.Length <= 0)
     {
         return;
     }

     //Set the minimum move up distance. If the new distance is smaller than the old distance, we don't bother putting it in. In other words, we're looking for the closest object. 
     float minMoveUpDist = float.MaxValue;
     foreach(RaycastHit hitInfo in hitInfos)
     {
         GameObject obstacle = hitInfo.collider.gameObject;
         minMoveUpDist = Mathf.Min(minMoveUpDist, hitInfo.distance);
     }

     //Now apply that distance to our camera.
     if(minMoveUpDist < float.MaxValue)
     {
         transform.position = rayStart + rayDir * minMoveUpDist;
     }

     return;

All of this should be working, but the camera is being set to my player's position. I'm wondering why, and I'm almost positive it's a layer mask problem. Here's the mask I use:

   m_RaycastHitMask = ~LayerMask.GetMask("Player", "Ignore Raycast");

It ignores everything except for the Player and Ignore Raycast layers. All of my game objects which shouldn't be hit are set to either of these, but it still registers the player as an obstacle. Any ideas on why this is?