Setting transform.up without losing lateral rotation based off gravity

I've created a player script for a ball that well rolls around as the player. It ray casts below it, gets the normal of the surface hit, and changes gravity to the normal * -9.81 for "realistic gravity". I then have a separate GameObject that is always lerps its transform.up to the new gravity (normalized of course) and there is a child of the gravity indicator that is used for lateral look around and then the camera is a child of that, which controls vertical look movements. So it's all based off that gravity indicator parent currently. It all works except one part. I noticed that especially when the indicator tries to be upside down it flips its X vector backwards. I understand that this is the behavior of the transform.up being set, it automatically adjusting itself to meet whatever vector was given, so I am trying to find a way to prevent this. My overall goal is to have a moveable camera that orientates to gravity and this is the closest I've gotten. I've searched some other ideas like using:

 Vector3 gravityAlongLocalZAxis = gravIndicator.transform.forward * 
                                  Vector3.Dot(augGravity, transform.forward);
 Vector3 gravityInLocalXYPlane = augGravity - gravityAlongLocalZAxis;
 gravIndicator.transform.LookAt (gravIndicator.transform.forward, -gravityInLocalXYPlane);

But that did something very strange and orientated the Z axis globally on the negative X axis and the YX plane vectors were rotating back and forth by 90 degrees. Any help is greatly appreciated!