Setting a RigidBody's velocity messes with my custom gravity, not sure how to proceed.

I'm trying to create my own Gravity system that's local to the player, where "down" gravity wise is where their feet are at, so that if you rotate the player 90 degrees they start falling towards the wall.

Issue is, when I'm setting the velocity to move the player around, if I set the Y value of the new velocity vector to rigidbody.velocity.y, then jumping is fine and gravity moves normally on the global Y axis, but when the player is rotated, they fall in the new direction(i.e. right, left, up, etc) , but much slower. If I set the Y value of the new velocity vector to 0, gravity works as I intended, but jumping is jerky and not smooth.Here's my entire movement script, sorry for the commented out code, I was trying other things: private Rigidbody playerRigidbody; public float acceleration; public float maxSpeed; public float jumpForce; public float slowDownSpeed = 1; private float raycastDistance = 1.5f; private Vector3 gravityVector; public float xGrav; public float yGrav; public float zGrav; void Start() { playerRigidbody = GetComponent<Rigidbody>(); gravityVector = new Vector3(0, -yGrav, 0); } void FixedUpdate() { Gravity(); Jump(); Movement(); } private void Gravity() { playerRigidbody.AddRelativeForce(gravityVector * Time.deltaTime); } private void Movement() { float horizontalMove = Input.GetAxis("Horizontal"); float verticalMove = Input.GetAxis("Vertical"); playerRigidbody.velocity = transform.TransformDirection(new Vector3(horizontalMove * acceleration * Time.deltaTime, playerRigidbody.velocity.y, verticalMove * acceleration * Time.deltaTime)); } private void Jump() { if (Input.GetKeyDown(KeyCode.Space)) { if (IsGrounded()) { playerRigidbody.AddRelativeForce(0, jumpForce, 0);// ForceMode.Impulse); } } } private bool IsGrounded() { return Physics.Raycast(transform.position, Vector3.down, gameObject.GetComponent<Collider>().bounds.extents.y + .1f); //raycastDistance); }