Problem with if statement overriding another with directional movement against terrain edges

I'm attempting to modify my character controller script to detect edges of terrain and stop movement if walking directly into an edge, and to slide along the edge if moving into it diagonally. All of the code for detecting the edges is working properly but I'm having an issue I can't seem to figure out with handling running into an edge on all directions. Tried many different ways of doing it and so far it always seems to come down to the same issue with the second directions if statement overriding the first on at least one of the four edges and causing it to either be jerky when sliding against the edge or go out of bounds. This is the most recent attempt at it and it handles the left/right edges perfectly and will stop if moving directly into the bottom or top edges but sliding against it causes the character to go off of the terrain.

With this code when you move against either the left or right edges dirZ remains at 0 and dirX stays at 1 as it should and the movement works properly, The issue is on the bottom or top edge the second if statement seems to get overridden by the first, when moving directly down into the bottom edge dirZ is set to 1 and movement stops but when pressing down and left/right it's changing dirZ to 0 and dirX to 1 causing it to go out of bounds. I'm sure it's something simple that I'm not considering here, any help would be appreciated.

 float dirX = 0f;
 float dirZ = 0f;
 Vector3 edgeMoveVector = new Vector3(0f,0f,0f);
 
 if ((direction.x > 0 || direction.x < 0) && dirZ == 0) dirX = 1;
 if ((direction.z > 0 || direction.z < 0) && dirX == 0) dirZ = 1;
 
 if (Mathf.Abs(dirX) > Mathf.Abs(direction.x))
     {
         edgeMoveVector += new Vector3(0f,0f,direction.z);
     }
 if (Mathf.Abs(dirZ) > Mathf.Abs(direction.z))
     {
         edgeMoveVector += new Vector3(direction.x,0f,0f);
     }