Custom Collision Detection With SweepTestAll

I hate collision detection. Its honestly the reason I started using Unity, and now I'm working on a project that because of... reasons... which I have researched thoroughly, I now have to write my own collision detection.

All of my objects are using AABB cubes. I am using SweepTest on my player to get potential collisions. I then call the function shown below on the RaycastHit SweepTestAll returns. (Note: I plan on using SweepTestAll and sorting the collisions to find the closest one that I want. The first line of the function I'm showing below is how I'm "sorting" right now but that is definitely working.)

The problem I'm having is that if I approach an object with which the player should collide from one side, he passes right through. From the other, he kind of jitters on its border.

Just a note, I call this function every FixedUpdate, which is also where I actually apply velocity to the transform of the player, and yes I do it after I call this function. (Sorry, I'm a little irritable right now.)

I need the player to stay .1f on any axis away from the collider because if he gets any closer, SweepTest wont return a hit. (Basically if they're already touching, it won't return a hit.) I've been struggling with this for two days now. Any fixes, or even a better way to do this would be absolutely wonderful!

     void HandleSweepTest(RaycastHit hitInfo){
         if(hitInfo.collider.gameObject.GetComponent<Collisions>().getCollisionType() == Collisions._CollisionType.solid){
 
             Vector3 relevantBounds = Vector3.Scale (collider.bounds.extents, (hitInfo.normal * -1));
 
             Vector3 transformedBounds = collider.transform.position + relevantBounds;
 
             Vector3 distToCol = hitInfo.point - transformedBounds;
             distToCol = Vector3.Scale (distToCol, hitInfo.normal);
 
             for(int i=0;i<3;i++){
 
                 if ((Mathf.Round(hitInfo.normal[i] * 100)/100) != 0){
 
                     float l = Mathf.Round (Mathf.Abs (distToCol[i]) * 100)/100;
                     if ((Mathf.Round (Mathf.Abs (distToCol[i]) * 100)/100) < .1f){
 
                         distToCol[i] = ((.1f * (distToCol[i]/Mathf.Abs (distToCol[i]))) + distToCol[i]);
                     }else if((Mathf.Round (Mathf.Abs (distToCol[i]) * 100)/100) > .1f){
 
                         distToCol[i] = distToCol[i] - (.1f * (distToCol[i]/Mathf.Abs (distToCol[i])));
                     }else{
 
                         distToCol[i] = 0;
                     }
                 }
 
                 distToCol[i] = Mathf.Round (distToCol[i] * 100)/100;
             }
 
             Vector3 distToMove = Vector3.zero;
             for (int i=0;i<3;i++){
 
                 if (hitInfo.normal[i] != 0){
 
                     distToMove[i] = distToCol[i];
                 }else{
 
                     distToMove[i] = velocity[i];
                 }
             }
             velocity = distToMove;
         }
     }