Help with 3D Boxcast

I'm pretty new to Unity. I am building an RTS style game for fun and am trying to get drag selection working properly.

I have an orthographic camera looking over the wold at an angle, and I draw the drag start / end using the GUI in screen coordinates. This works.

I then figure out the 4 world points for each of the box corners using Camera.main.ScreenToWorldPoint.

 Vector3 wp1 = Camera.main.ScreenToWorldPoint (screenPos1);
 Vector3 wp2 = Camera.main.ScreenToWorldPoint (screenPos2);
 Vector3 wp3 = Camera.main.ScreenToWorldPoint (new Vector2(screenPos1.x, screenPos2.y));
 Vector3 wp4 = Camera.main.ScreenToWorldPoint (new Vector2(screenPos2.x, screenPos1.y));

Using these 4 points I can compute the center, the extents, etc.

 Vector3 wpcenter = Camera.main.ScreenToWorldPoint (
     new Vector2(screenPos1.x + (screenPos2.x - screenPos1.x) / 2f,
             screenPos1.y + (screenPos2.y - screenPos1.y) / 2f)
 );

 Vector3 halfExtents = new Vector3(wpcenter.x - wp1.x, wp1.y - wpcenter.y, 25);

I then do a box cast into the world from the camera to see what I'm selecting:

 Vector3 direction = Camera.main.transform.forward;
 Quaternion rotation = Quaternion.LookRotation (direction);
 RaycastHit[] hits = Physics.BoxCastAll (wpcenter, halfExtents, direction, rotation, Mathf.Infinity);

This does sort of work, but I have something wrong and I can't really figure it out. If I drag the selection box really wide or really tall it doesn't select what's under the GUI box.

To help visualize the box cast I create a cube and manually set the vertices to be the 4 world points above and position it in front of the camera. Ideally this would line up perfectly with the GUI selection box, but clearly it does not.

         if (_boxCastVisual == null) {
             _boxCastVisual = GameObject.CreatePrimitive (PrimitiveType.Cube);
             _boxCastVisual.GetComponent<MeshRenderer> ().material.color = Color.yellow;
         }
         _boxCastVisual.transform.transform.position = wpcenter;
          _boxCastVisual.transform.localScale = new Vector3 (halfExtents.x * 2, halfExtents.y * 2, 4);  
         _boxCastVisual.transform.rotation = rotation;

         // move it foward so we can see it
         _boxCastVisual.transform.Translate (Vector3.forward * 100);

The result can be seen in these screenshots:

https://www.dropbox.com/s/we7yufm0uv7mz2m/Screenshot%202016-06-08%2007.54.31.png?dl=0 https://www.dropbox.com/s/sg59pfd1mlb0bhd/Screenshot%202016-06-08%2007.54.36.png?dl=0 https://www.dropbox.com/s/jl8kpph3p3tir55/Screenshot%202016-06-08%2007.54.42.png?dl=0

Can someone point to my error? Thanks!