Collider not continuing to Ignore CharacterController after center changes.

So basically I have an Item(GameObject with rigidbody and boxcollider) which ignores collisions with my CharacterController, currently it works as intended, however if my CharacterControllers' center changes during runtime,(which it does when I crouch) the Item no longer ignores collisions with the Controller. I would like the collision to be ignored all the time. Maybe I'm just having a brain fart day, so I'm hoping one of you wonderful members of the Unity Community could help me, or point me in the right direction.

Here is the Crouch part of my CharacterMotor.

 if(crouching.enabled)
         {
             mainCamera.transform.localPosition = crouching.cameraPos;
             if(inputCrouch == true)
             {
                 crouching.holdingCrouchButton = true;
                 if(crouching.holdingCrouchButton)
                 {
                     controller.height = Mathf.Lerp(controller.height,crouching.crouchHeight,crouching.crouchSpeed);
                     controller.center = Vector3.Lerp(controller.center,crouching.crouchCenter,crouching.crouchSpeed);
                     jumping.enabled = false;
                     crouching.cameraPos.y = Mathf.Lerp(crouching.cameraPos.y,crouching.cameraHeight-1,crouching.crouchSpeed);
                     sprinting.sprinting = false;
                     crouching.crouching = true;
                 }
             }
             else
             {
                 if(controller.center != defaultControllerCenter)
                 {
                     controller.center = defaultControllerCenter;
                 }
                 if(controller.height != defaultControllerHeight)
                 {
                     controller.height = defaultControllerHeight;
                 }
                 jumping.enabled = true;
                 crouching.crouching = false;
                 crouching.cameraPos.y = Mathf.Lerp(crouching.cameraPos.y,crouching.cameraHeight,crouching.crouchSpeed);
             }
         }

And here Is the part of my Item Script which Ignores the controller.

 void Start()
     {
         if(Player != null)
         {
             Debug.Log("PlayerFound");
         }
         Physics.IgnoreCollision(Player.GetComponent<CharacterController>(),collider);
     }

I am avoiding Using IgnoreCollisionLayer, because of future reasons. And I have tried to ignore the collision during an Update function, however it doesn't continue to ignore while the controller is crouched, only when it returns to its standing position. Any help would be greatly appreciated.