Need help camera relative movement, script works but if camera is on z values need to be positive, if camera is on x values need to be negative

Hey guys I'm a pretty amateur coder and I've been looking for a simple camera relative movement controller and this is as close as I've come so far, it keeps the camera position relative to the player while looking at a pivot that is set to the players position.

my problem is when i have the cam on the world z axis (behind or in front of character) positive values work for relative movement but if i use positive values with my camera on the world x axis the player moves in the opposite direction to the input relative to the camera.

the problem appears to be in the last two lines of code.

if someone could explain to me what i could do to fix this and why its happening it would help me greatly.

using UnityEngine; using System.Collections;

public class RelativeMovement : MonoBehaviour {

 //camComponents
 Camera cam;
 Transform pivot;
 Vector3 offset;

 //playerComponents
 Animator anim;
 GameObject player;
 Rigidbody rb;
 float walkSpeed = 4.0f;

 //inputComponents
 public float horizontal;
 public float vertical;

 void Start () {
     anim = GetComponent<Animator> ();
     cam = Camera.main;
     player = GameObject.FindGameObjectWithTag ("Player");
     rb = GetComponent<Rigidbody> ();
     pivot = cam.transform.parent.transform;
     offset = transform.position - cam.transform.position;
 }
     
 void Update () {
     InputMoveFindCameraPos ();
     if (Input.GetKeyDown("e")){    anim.Play ("RightHook");    }
 }

 void LateUpdate(){
     cam.transform.position = transform.position - offset;
     cam.transform.LookAt (pivot);
 }

 void InputMoveFindCameraPos(){
     
     horizontal = Input.GetAxis ("Horizontal");
     vertical = Input.GetAxis ("Vertical");
     Vector3 moveDir = new Vector3 (horizontal, 0, vertical);

     if (moveDir != Vector3.zero) {
         //Move pivot with player and position it up for a better view of character
         pivot.position = transform.position + Vector3.up;

         //find the direction the camera is looking in and use that with input
         Vector3 relativeToCam = cam.transform.InverseTransformDirection (moveDir);
         // set the y difference to 0 so player doesn't move away from the ground
         relativeToCam.y = 0.0f;

         // this is where the problem is 
         // these next two lines work but using them 
         // as positive works: when camera is on z axis
         // as negative works: when camera is on x axis 
         transform.rotation = Quaternion.LookRotation (-relativeToCam); 
         rb.MovePosition (transform.position - (relativeToCam * walkSpeed * Time.deltaTime));
     }
 }

}