koobas.hobune.stream
'Invert' UI mask
Hows it going all,
I am working on some transitions for my game, and one of the ideas that came to mind was a keyhole transition where a shape is cutout in solid color to reveal what is behind and that hole slowly becomes smaller to cover the screen. To do this I need an inverted mask so that the shape I put in it cuts out from the parent image, I have found some threads with shaders that do not work in unity 5, any suggestions?
I'm trying to fix the same problem. So I come up to the idea to switch stencil operation.
here is a code
using UnityEngine;
using UnityEngine.Rendering;
using UnityEngine.UI;
public class InvertedMaskImage : Image {
public override Material materialForRendering
{
get
{
Material result = base.materialForRendering;
result.SetInt("_StencilComp", (int)CompareFunction.NotEqual);
return result;
}
}
}
and this is the scene setup
Use the same Mask object on the MaskSprite. but for CoverScreenSprite change Image component to the InvertedMaskImage. so it will replace stencil comparison operation.
Unfortunatly inspector will still show old material parameters, 
but it will work. in editor as well as in play mode
Hi, I used your code and it broke my original UI image, you have to instiate for it to work without conflicting with the current one : using UnityEngine; using UnityEngine.Rendering; using UnityEngine.UI;
public class UIInvert$$anonymous$$ask : Image
{
public override $$anonymous$$aterial materialForRendering
{
get
{
$$anonymous$$aterial result = Instantiate( base.materialForRendering);
result.SetInt("_StencilComp", (int)CompareFunction.NotEqual);
return result;
}
}
}
It's an old question but I noticed a unwanted sideeffect with nicloay's solution in Unity 2019.4.x. The script will modify your default ui material stencil setting (all default sprites will become invisible).
Here is my approach without coding which is similar to nicloay's solution but uses one dedicated material for mask inversion. It also avoids creating / instancing new materials.
If you don't want to touch any shaders, there are 3 ways of doing this.
One easy, but the results might be bad (depending on many factors) and one that is somewhat safe, but will require some work on your part to allow it to appears as intended on the screen. I'll explain both ways with, as an example, your idea of a keyhole where the hole is the invisible part of the UI that display the scene or whatever is behind it.
The easy way of doing this is to only use a inverted texture as a UI mask. By using an "huge" UI Image that exceed on all side of the screen, you could have your keyhole set in that.
This is a quick visual reference : 
The issue with this method is that the hole can't be crystal clear. This means it can work if you're using a smoothed or blurry keyhole right from the start. (By "Resize the Texture", I meant the UI RectTransform of the Image component that hold the keyhole Sprite.)
The other method that doesn't involve making your own or using someone else shader is to do a small trick where you use a much clearer (and possibly smaller) image file of the keyhole inverted. It's like the first option above, but instead of using a HUGE amount of "filler" around the keyhole, you use a set of 3 row and 3 columns where the keyhole is kept in the middle. You can do this by using an intelligent set of Layout groups and uses of Layout Element components. After, you just got to adjust the middle keyhole's RectTransform sizes while letting the 8 surrounding "plain" colored UI element readjust themselves.
The last method requires a bit of work, yet still doesn't need any kind of special shaders. You can use an additional camera with and a higher priority (larger depth) and with Depth Only for its Clear Flag which only see the keyhole texture/mesh (through its culling mask) and animate/move the camera into or out of the the keyhole. This method is a lot more useful if you plan to use some 3D modeled keyhole or if you want more control (via script) over the keyholes animations and/or events. The only issues you might accounter with this last/3rd method is memory/vram wise. Adding a camera for such might be a problem if you're aiming at some light-weight smartphones or tablet as such each camera requires its own Z-Buffering channel to "cut" out the required stuff and put each "results" over each other in the right order.
