How is the Bounds.SqrDistance calculated behind the scene? What algorithm/formula does it use?

Well, the actual implementation is hidden in the native code of Unity. However since the Bounds is an AABB (axis aligned bounding box) that's actually quite simple. You just need to calculate the closest point on the surface of the bounds and then calculate the distance from that point. To get the closest point you just need to clamp the 3 components of the point you want to know the distance to against the min / max values of the bounds for each axis. Sounds complicated but it isn't.

So if you have min and max of your AABB as two vectors you get the projected point like this

 float3 pointOnBounds = clamp(point, min, max);

In C# would would need to do 3 seperate clamp calls for each component. In a shader we can actually do this with a single line. So this line in HLSL is equivalent to this C# code:

 Vector3 pointOnBounds;
 pointOnBounds.x = Mathf.Clamp(point.x, min.x, max.x);
 pointOnBounds.y = Mathf.Clamp(point.y, min.y, max.y);
 pointOnBounds.z = Mathf.Clamp(point.z, min.z, max.z);

Once you have the point on the bounds you can simply calculate the distance by subtracting your point from it

 float3 dir = pointOnBounds - point;

To get the squared distance that the direction vector represents, you just need to get the dot product with itself

 float sqrDistance = dot(dir, dir);

or if you actually want to know the distance, you could directly use the distance method

 float dist = distance(pointOnBounds, point);

Note: if your point is outside the AABB you get a distance / sqrDistance that is greater than 0. If the point is inside the AABB the distance would always be 0 since the projected point would always be itself.

SDF shaders use this equation:

 // src* https://www.iquilezles.org/www/articles/distfunctions/distfunctions.htm
 float sdBox( float3 p, float3 b )
 {
   float3 q = abs(p) - b;
   return length(max(q,0.0)) + min(max(q.x,max(q.y,q.z)),0.0);
 }

note:

returns signed distance

b(ox half size) is aabb.extents (i.e. aabb.size*0.5f )

p(oint) is relative to aabb.center