How to create User layer that acts like the "Ignore Raycast" layer?

Im trying to create another layer that acts exactly like the built-in layer "IgnoreRaycast". Ive been trying to figure out how to correctly do this but still not sure how. I have two layers Ive created that I would like to act as Ignore Raycast layers. The layers I would like to ignore are layers 20 & 21. The layer "unitmask" is picked inside the inspector when looking at the character.

Below: OnMouseOver works correctly but when leaving the character's "unitmask" the OnMouseExit doesn't work because it is hitting the layers I want to ignore. How can I Ignore layers 20 & 21?

 public LayerMask unitmask;
 public int IgnorelayerF = ~(1<< 20);
 public int IgnorelayerE = ~(1<< 21);
         
         public void OnMouseOver(){
         
                 Ray ray = Camera.main.ScreenPointToRay (Input.mousePosition);
                         if (Physics.Raycast (ray, out hit, Mathf.Infinity, unitmask)) {
                         
                     print ("Hit unit mask");            
     } 
 }
             
             public void OnMouseExit(){
         
                 Ray ray = Camera.main.ScreenPointToRay (Input.mousePosition);
                         if (Physics.Raycast (ray, out hit, Mathf.Infinity, IgnorelayerE) || Physics.Raycast (ray, out hit, Mathf.Infinity, IgnorelayerF)) {
                     
                     print ("Not hitting unitmask");
                         
         }
 }