2 Player Splitscreen w/ Xbox Controllers. Player 1 gets inverted controls when facing a different direction

Let me set the scene for everyone:

1. Unity FPS Controller Scripts with public variables naming their axis so I could reuse the same script

2. Player 1 is Joystick 1

3. Player 2 is Joystick 2

4. Player 2 can move around perfectly fine, look around, jump etc

5. Player 1 can only look, move and jump when he is facing a certain direction. If I turn around and try to move his controls become inverted.

Things I've Tried so far:

1. Deleting P1 and Creating a new one. The problem now switches to Player 2 instead of Player 1.

2. Copying the Scripts and renaming them. Same problem happens.

Anyone have any thoughts because I'm stuck.