Collisions in Custom Physics Using Circlecast

I am making a 2D game heavily based on Worms Armageddon. I want to implement my own physics instead of using Rigidbody2D so I can have much greater control over how things behave and when they should collide and when they shouldn't, or make only one object react to a collision instead of both, exc. In my experience rigidbody is much harder to tame than just using your own code. Please do not tell me to use rigidbody.

I have a problem with getting my physics to correctly know when to settle or slide along a surface and when to bounce. When the y velocity gets low enough it ends up "jittering" along the ground and cannot come to a rest. Sometimes it also 'sticks' to a surface when it shouldn't. My physics uses circle casts only. I believe this to be good enough for all situations. I'm using polygon colliders for my static terrain.

I have spent two full, solid days trying to get this working the way I want and I'm completely lost. Help and/or code snippets would be a massive help and a relief.

I apologize for the messy code. I have tried a lot of different things.

 public float maxVelocity = 32f;
 float time; //time in ticks, a.k.a. frames (50ths of a second)
 public float speed = 1f; //Speed to run the simulation
 public float wind = 0f;        //wind (pixels per tick per tick)
 public float grav = -0.24f;     //gravity (pixels per tick per tick) def: 0.24
 public float radius = 0.08f;    //Radius of our "circle collider"
 public float bounceMultX = 0.96f; //Multiply velocity by this much when hitting something horizontally
 public float bounceMultY = 0.96f; //Multiply velocity by this much when hitting something vertically
 public LayerMask layerMask;
 public float sleepSpeed = 0.01f; //Speed below which we will begin to sleep.
 public float penetrationError = 0.01f; //Push ourselves out by this much so next frame our next circle cast will not hit what we already collided with.
 public Vector2 cVel = Vector2.zero; //Current velocity
 Vector2 cPos = Vector2.zero;        //Current position
 void Update()
 {
     cPos = new Vector2( transform.position.x, transform.position.y );
     time = Time.deltaTime * speed;
 
     if( asleep == false )
         PhysicsUpdate();
 }
 public bool grounded = false;
 public bool asleep = false;
 
 private Vector2 lastNormalHit = Vector2.zero;
 private Collider2D lastCollider;
 private Vector3 lastPosition = Vector3.zero;
 public Collider2D ignoredCol;
 void PhysicsUpdate()
 {
     //'c' = 'current'
     //'n' = 'new'
     float w = wind;//(lastCollider == null) ? wind : 0f;
     float g = grav;//(lastCollider == null) ? grav : 0f;
 
     Vector2 nPos; //http://worms2d.info/User:Etho/Physics_Engine
     nPos.x = cPos.x + (cVel.x * time) + (w * time * (time + 1f)) / 2f; 
     nPos.y = cPos.y + (cVel.y * time) + (g * time * (time + 1f)) / 2f;
     Vector2 nVel;
     nVel.x = cVel.x + (w * time);
     nVel.y = cVel.y + (g * time); //These equations give a nice floaty feel and give predictable arcs
 
     Debug.DrawLine( transform.position, new Vector3( nPos.x, nPos.y, 0f ), Color.red );
 
     //Check for collisions between cPos and nPos. Override position and/or velocity if so
     CollisionCheck( ref nPos, ref nVel );
 
     if( Mathf.Abs(nVel.x) < sleepSpeed )
         nVel.x = 0f;
     if( Mathf.Abs(nVel.y) < sleepSpeed )
         nVel.y = 0f;
 
     cVel = Vector3.ClampMagnitude( nVel, maxVelocity ); //TODO clamp velocity
     transform.position = new Vector3( nPos.x, nPos.y, 0f );
     
     lastPosition = transform.position;
 }
     
 void CollisionCheck( ref Vector2 tPos, ref Vector2 fromVel )
 {
     Vector2 fromDir = VectorExtras.Direction(cPos, tPos);
     float targetDist = Vector2.Distance(cPos, tPos);
 
     RaycastHit2D data = CircleCast(cPos, radius, fromDir, targetDist, layerMask);
     if( data == false )
     {
         lastNormalHit = Vector2.zero;
         lastCollider = null;
         return;
     }
 
     if( lastNormalHit != Vector2.zero )
     {
         if( Mathf.Abs(Vector2.Dot( lastNormalHit, data.normal )) < 0.8f )
         {
             Debug.Log("Collision skip");
             lastNormalHit = data.normal;
             lastCollider = data.collider;
             return;
         }
 
     } 
     Vector2 impactPoint = data.centroid;//VectorExtras.OffsetPosInDirection(data.centroid, data.normal, penetrationError);
     float remainingDist = targetDist - Vector2.Distance(cPos, impactPoint);
 
     Vector2 nVel = Vector2.Reflect(fromVel, data.normal);
     
     //Move the remaining distance
     Vector2 nPos = VectorExtras.OffsetPosInDirection(
         impactPoint, //Push impact point out a bit
         nVel.normalized, 
         remainingDist );//nVel.magnitude * time * data.fraction ); 
 
 
     Draw(nPos, Color.white);
     //Check to make sure we aren't trying to move further into the ground
     RaycastHit2D dataNewPos = CircleCast(nPos, radius, Vector2.zero, 0f, layerMask);
     if( dataNewPos == true )
     {
         //New position was not valid! Do a raycast opposite of data.normal to determine where the surface actually is, then bounce from there instead.
         Vector2 errPnt = VectorExtras.OffsetPosInDirection(data.centroid, data.normal, radius);
         Vector2 errEnd = VectorExtras.OffsetPosInDirection(errPnt, -data.normal, radius * 2f);
         Debug.DrawLine( new Vector3( errPnt.x, errPnt.y, 0f ), new Vector3( errEnd.x, errEnd.y, 0f ), Color.cyan, 1f );
 
         RaycastHit2D dataSurfaceHit = CircleCast( errPnt, radius + penetrationError, -data.normal, radius * 2f, layerMask);
         
         if( dataSurfaceHit == false )
             Debug.LogWarning("Error correction failed!");
         else
         {
             nPos = VectorExtras.OffsetPosInDirection(dataSurfaceHit.centroid, dataSurfaceHit.normal, penetrationError);
 
             if( Mathf.Abs(grav * time) > targetDist )
             {
                 nVel = ProjectOnLine( nVel, dataSurfaceHit.normal );
                 grounded = true;
                 Debug.Log("Grounded");
             }
             else
             {
                 Vector2 reflect = nVel;
                 Vector2 bouncePush = VectorExtras.Direction( dataSurfaceHit.centroid, nPos ) * nVel.magnitude;
                 //Lerp between the velocities to blend them
                 nVel = Vector2.Lerp( reflect, bouncePush, 0.5f );
             }
 
             nPos = VectorExtras.OffsetPosInDirection( nPos, nVel.normalized, remainingDist );
 
             Debug.DrawLine(new Vector3(dataSurfaceHit.centroid.x, dataSurfaceHit.centroid.y, 0f), new Vector3(nPos.x, nPos.y, 0f), Color.magenta, 1f);
             Debug.Log("Surfaced");
         }
     }
 
     if( bounceMultX != 1f )
     {
         nVel.x = nVel.x * bounceMultX;
     }
     if( bounceMultY != 1f )
     {
         nVel.y = nVel.y * bounceMultY;
     }
 
     Debug.DrawLine(new Vector3(data.centroid.x, data.centroid.y, 0f), new Vector3(nPos.x, nPos.y, 0f), Color.green, 1f);
 
     tPos = nPos;
     fromVel = nVel;
     lastNormalHit = data.normal;
     lastCollider = data.collider;
 
     SendMessage("OnCollisionEnter2D", new Collision2D(), SendMessageOptions.DontRequireReceiver);
 }

Here are images of the problem:

https://gyazo.com/adfc020068572b4adb5aae435c548f3e

https://gyazo.com/c78149aeee1d66304c30779a0df96543

Unrelated img of the game so far: https://gyazo.com/c705f9de51f6bc6990e8b0a222fdcd53