How do I use layermasks?

This is an attempt to simplify Stelimars insightful post.

LayerMask, or any "Mask" in Computer Science, is an integer which uses its 32 bits to represent different flags. While it is an integer, the value doesn't mean anything useful numerically. What matters is which flags are turned on or off.

In the case of LayerMask, each flag (each bit) represents one layer. If you select any game object from the editor you'll notice the Layer field in the top right corner of the inspector. Clicking this will bring up a list of different layers. There are some predefined Layers, but you can also add layers. Notice how you have 32 slots for your layers. This is because Unity uses an integer type to store the flags in each of the 32 bits that makes up an integer. As we can see, Layer 8 to 31 is available for us to define for our own game.

Since Masks use a single bit to represent if some option is true or false, a Mask can contain many different options. The same goes when thinking about Layers. A LayerMask specifies which Layers are enabled for any given mechanism. This enables us to tell the game that our GameObject "Don", for example, is thought to belong to the user defined layer "NPC" and "Bob" to belong to the user defined layer "SpellCaster".

Then, we make us of this information for example in a raycast call, specifying that the raycast should only return objects that belong to the "NPC" layer. An example when this could be used is when you want to target NPCs only with the click of the mouse. Then, imagine you have a game where you want to cast a spell, and that spell can only target Spell Casters. Then you'd only include the game objects that has the SpellCaster layer enabled. A third option would be to include both NPC and SpellCaster in the search. That way we would test both Don and Bob.

Finally some technical mumbo.

Technical

---

A LayerMask is represented as a 32bit integer. It also has a name table for naming the different bits so it is more readily understood by us game developers. Just to give some perspective, here's how a 32 bit integer would look like if it was written.

0000 0000 0000 0000 0000 0000 0000 0000

I put a space between every 4th bit to make it easier to read.

The first bit is actually located on the right side. Think of this as being the first layer. Here's how it would look if the first bit (or layer) would be set:

0000 0000 0000 0000 0000 0000 0000 0001

Since Unity predefines 8 layers, we can display the Mask for including all of those layers as such:

0000 0000 0000 0000 0000 0000 1111 1111

And if we like to search for NPC as from the example above, given we've put it in User Layer 8 (starting from 0, this is the 9th layer), we'd mask it as:

0000 0000 0000 0000 0000 0001 0000 0000 // NPC Layer

Just for completeness, this is how you should think a mask that include NPC OR SpellCaster should look like, if we assume the SpellCaster was set to User Layer 9:

0000 0000 0000 0000 0000 0011 0000 0000 // NPC | SpellCaster Layer

Example

---

Allright, now that you see we're setting bits to enable the different layers, let's look at how we generate the bit field from scratch using C#. In my example, I will make three masks that can be used to query NPCs, SpellCasters, and "NPCs and SpellCasters". We'll use Physics.Raycast in our example.

Example 1: Raycast vs NPCs

 int NPCLayer = 8; 
 // This is used to illustrate which Layer we want to use.
 
 int NPCMask = 1 << NPCLayer; 
 // This is used to generate the Mask derived from NPC Layer.
 
 if Physics.Raycast(fromPosition, direction, Mathf.Infinity, NPCMask)
 {
     // We cast a ray onto a NPC!
 }

Example 2: Raycast vs SpellCasters

 int SpellCasterLayer = 9; 
 // This is used to illustrate which Layer we want to use.
 
 int SpellCasterMask = 1 << SpellCasterLayer; 
 // This is used to generate the Mask derived from SpellCaster Layer.
 
 if Physics.Raycast(fromPosition, direction, Mathf.Infinity, SpellCasterMask)
 {
     // We cast a ray onto a spell caster!
 }

Example 3: Raycast vs NPCs and SpellCasters

 int NPCLayer = 8; 
 int SpellCasterLayer = 9; 
 // This is used to illustrate which Layers we want to use.
 
 int NPCMask = 1 << NPCLayer; 
 int SpellCasterMask = 1 << SpellCasterLayer; 
 // This is used to generate the Mask derived from the layers.
 
 int CombinedMask = NPCMask | SpellCasterMask;
 // We use the binary OR operator to combine masks.
 
 if Physics.Raycast(fromPosition, direction, Mathf.Infinity, CombinedMask)
 {
     // We cast a ray onto a NPC or a SpellCaster!
 }

Stelimar also have excellent information in his posts that you should read.

     public int npcComp = 8;
     public int playerComp = 9;
     public int terrain = 10;
     
     public Layer$$anonymous$$ask hittableLayers; //public just for easy display.
 
 
     void Start()
     {
         hittableLayers = (1 << npcComp) | (1 << playerComp);
         hittableLayers = hittableLayers | (1 << terrain); //'adds' terrain layer.
         hittableLayers = hittableLayers ^ (1 << terrain); //'removes' terrain layer.
     }

         hittableLayers = hittableLayers & ~(1 << terrain); //'removes' terrain layer.

While the information about layermasks in the other answers is good, it would be remiss not to mention the LayerMask type. Generally this is the easiest way to use them, since you don't have to convert anything or know anything about binary math (although it's a good idea to have that knowledge anyway, in case you want to do more complex layermask coding).

 var myLayerMask : LayerMask;
 
 if Physics.Raycast(fromPosition, direction, Mathf.Infinity, myLayerMask)
 {
     // Do something
 }

This way you can simply choose the appropriate layers in the inspector.

The answers above do a great job of explaining how to use LayerMasks and the technical stuff behind it. This is just a little additional information I thought people might find helpful.

I find that working with LayerMasks by name instead of layer number makes the process a lot cleaner to read.

The LayerMask class has static methods

 int NameToLayer(string layerName)   // Returns Layer Number From Layer Name
 
 string LayerToName(int layer)       // Returns Layer Name From Number

When I build LayerMasks in my scripts, I alway use the names instead of the numbers. For instance...

 lightBeamLayerMask =  (1 << LayerMask.NameToLayer("World Geometry")) 
                     | (1 << LayerMask.NameToLayer("Light Triggers"));

This way I don't have to worry about whether "World Geometry" is Layer 8, or 10.

Also, I usually store LayerMasks as private variables and build them once in the Start() method, rather than creating them immediately before using them like the Unity Scripting Reference Examples.

Although you can always use the other mentioned methods to get the perfect layer-mask you desire. I'd like to point out another possibility to have the layermask. Which is actually putting the integer hardcoded. It's not recommended though, unless you've got some problems with any other method, or may be if you've a doubt that your bitwise calculations are not correct.

For eg. if you want to include layer 2 and 4 in the layermask, the calculation should be

 int mask = 1<<2 | 1<<4;

which will result into 0001 0100 which in decimal results into 20, so you can just put 20 as a layermask too.

 int mask = 20;

So instead of putting the calculation in the code, you are doing calculations yourself and putting just the result. However since it's hardcoding, I'd not recommend using this method except for debugging.