Find best path to closest edge of model

Edit: See TL;DR below.

I'm using the A* Pathfinding project to move a unit from point A to a building, using the buildings transform.position. This works, and consistently the algorithm comes up with the same path. Unfortunately, that path for some reason wanders through some obstacles to approach the building from the FAR side of it's model. It would produce a shorter path if I could give it a target Vector3 on the closer edge of the model.

I believe I could do this easily by casting a ray from my unit to the building model, and returning the point at which I hit the model, but that point is not necessarily always the closest point pathing wise, though it is in this particular case.

Has anyone encountered this before? Is there a best practices I should be aware of for this situation?

I will eventually need to do pathing for 1-200 units at a time, so I'm hoping to avoid testing paths against all four corners of the building, or something like that.

Thanks!

TL;DR: I think I can simplify this question.

Picture a building with a transform.position of (5,0,5). The actual dimensions of the building are 10x0x10, meaning the bottom left corner is at 0,0,0 and the top right corner is at 10,0,10. I'm using the transform.position as the target coordinates, but there is a collider (Building model) at a minimum 5 units away, and at worst, roughly 7 units at the corners.

I would like to determine the closest path to ANY part of the building, not necessarily the center. This image might help.