Object swaying to side on slope

I'm developing a sliding mechanic, but I've noticed that when I slide down a hill, my player object sways to the left or right (presumably depending to which side it leans more). A similar issue that I have is when I'm standing still after walking and my character heading to one side of the slope (again presumably depending to which side it leans more) slowly. I've checked, and there seems so be no forces being added to the RB to cause this, by the movement script at least (which is currently the only script that adds velocities).

I tried to the velocity when it's heading towards the side, but seeing as I intend on having ramps heading different directions, I can't think of a possible way to pinpoint the 'sideways' axis, and then modify or limit the velocity of the player relative to this axis, seeing as in some cases it won't be a simple X or Z axis.

My code is pretty big, so I'll just upload the main sliding part. Here's the link to the entire code (Slide part is at line 139 - 200): https://code.sololearn.com/cgcRsLMkWE7W

 /*Sliding Movement*/ {
             if(Sliding == true) {
                 Vector3 SlidePosTemp = this.transform.position;
             
                 if(SlidePosTemp.y != SlidePos.y) {
                     SlideAngle = Vector3.Angle(transform.up, (SlidePosTemp - SlidePos).normalized) - 90;
                 }
                 
                 SlidePos = SlidePosTemp;
             }
                 
             float SlideVelTemp = CharVel.magnitude;
             
             if (SlideVelTemp > SlideVel + 5) {
                 this.GetComponent<Rigidbody>().velocity *= SlideVel / SlideVelTemp;
             }
             
             SlideVel = SlideVelTemp;
             
             
             if(Input.GetKeyDown(KeyCode.LeftControl) && SlideTime == 0) {
                 SlideTime = BaseSlideTime;
                 CanSlide = true;
                 Sliding = true;
             }
         
             if(Input.GetKey(KeyCode.LeftControl) && CanJump == true && CanSlide == true && Sliding == true) {
                 if(SlideTime > 0 || CharVel.magnitude > WallRunSpeedMin) {
                     Sliding = true;
                     
                     if(CharVel.magnitude < BaseSpeed) {    
                         this.GetComponent<Rigidbody>().drag = 0;
                     
                         if(this.transform.localScale.y == SlideHeight) {
                             this.GetComponent<Rigidbody>().AddForce((CharVel.normalized * SlideSpeed) * (1 + ((SlideAngle / 90) * SlideAngleAcc)));
                         }else if(this.transform.localScale.y > SlideHeight){
                             this.GetComponent<Rigidbody>().AddForce(CharVel.normalized * SlideSpeed);
                         }
                     }
                     if(this.transform.localScale.y > SlideHeight) {
                         this.transform.localScale -= new Vector3(0, 1.5f * Time.deltaTime, 0);
                         this.transform.position -= new Vector3(0, 1.5f * Time.deltaTime, 0);
                     }
                     if(this.transform.localScale.y < SlideHeight) {
                         this.transform.localScale = new Vector3(1, SlideHeight, 1);
                     }
                 }
             }
                 
             if(Input.GetKeyUp(KeyCode.LeftControl)) {
                 Sliding = false;
             }
             
             if(Sliding == false) {
                 if(this.transform.localScale.y < 1) {
                     this.transform.localScale += new Vector3(0, 1.5f * Time.deltaTime, 0);
                 }
                 if(this.transform.localScale.y > 1) {
                     this.transform.localScale = new Vector3(1, 1, 1);
                 }
             }
         }