Wayback Machinekoobas.hobune.stream
May JUN Jul
Previous capture 12 Next capture
2021 2022 2023
1 capture
12 Jun 22 - 12 Jun 22
sparklines
Close Help
  • Products
  • Solutions
  • Made with Unity
  • Learning
  • Support & Services
  • Community
  • Asset Store
  • Get Unity

UNITY ACCOUNT

You need a Unity Account to shop in the Online and Asset Stores, participate in the Unity Community and manage your license portfolio. Login Create account
  • Blog
  • Forums
  • Answers
  • Evangelists
  • User Groups
  • Beta Program
  • Advisory Panel

Navigation

  • Home
  • Products
  • Solutions
  • Made with Unity
  • Learning
  • Support & Services
  • Community
    • Blog
    • Forums
    • Answers
    • Evangelists
    • User Groups
    • Beta Program
    • Advisory Panel

Unity account

You need a Unity Account to shop in the Online and Asset Stores, participate in the Unity Community and manage your license portfolio. Login Create account

Language

  • Chinese
  • Spanish
  • Japanese
  • Korean
  • Portuguese
  • Ask a question
  • Spaces
    • Default
    • Help Room
    • META
    • Moderators
    • Topics
    • Questions
    • Users
    • Badges
  • Home /
avatar image
0
Question by tanoshimi · Oct 13, 2016 at 03:16 PM · stencilimage effect

Stencil Buffer in Image Effect not updating?

I'm trying to make use of the stencil buffer to selectively apply an image effect to pixels in a post-process image effect. The problem I'm having is that the stencil buffer doesn't seem to be getting properly cleared/re-populated, or perhaps OnRenderImage is not getting called, which means that the stencil mask I create never gets changed if the frame changes after the effect is first applied.

I've created a simple reproduction case of the problem, which can be achieved as follows:

Create a shader that writes a value "5" to the stencil buffer for every pixel rendered:

 Shader "Write5ToStencil" {
     SubShader {
         Pass {
             Stencil {
                 Ref 5
                 Pass Replace
             }
             CGPROGRAM
             #pragma vertex vert
             #pragma fragment frag
             float4 vert (float4 vertex : POSITION) : SV_POSITION{
                 return mul(UNITY_MATRIX_MVP, vertex);
             }
             fixed4 frag () : SV_Target {    
                 return fixed4(0.5, 0.5, 1, 1);
             }
             ENDCG
         }
     }
 }

Create a material that uses this shader, and place it on a few objects in the scene.

Now create another shader that will be used as an image effect. It'll have two passes that will test the value in the stencil buffer and draw white if there is something, or black if not:

 Shader "Hidden/Is5InTheStencilBuffer" {
     SubShader {
         // If the stencil buffer value is NOT 5, draw black
         Pass {
             Cull Off ZWrite Off ZTest Always
             Stencil {
                 Ref 5
                 Comp NotEqual
             }
             CGPROGRAM
             #pragma vertex vert
             #pragma fragment frag
             float4 vert (float4 vertex : POSITION) : SV_POSITION{
                 return mul(UNITY_MATRIX_MVP, vertex);
             }
             fixed4 frag () : SV_Target {    
                 return fixed4(0, 0, 0, 1);
             }
             ENDCG
         }
         // If the stencil buffer value IS 5, draw white
         Pass {
         Cull Off ZWrite Off ZTest Always
             Stencil {
                 Ref 5
                 Comp Equal
             }
             CGPROGRAM
             #pragma vertex vert
             #pragma fragment frag
             float4 vert (float4 vertex : POSITION) : SV_POSITION{
                 return mul(UNITY_MATRIX_MVP, vertex);
             }
             fixed4 frag () : SV_Target {    
                 return fixed4(1, 1, 1, 1);
             }
             ENDCG
         }
     }
 }

Create a material from this shader, and use it in a Blit() of the OnRenderImage function of a script attached to the camera, just like any regular image effect:

 [ExecuteInEditMode]
 public class ApplyImageEffect : MonoBehaviour {
     public Material mat;
     void OnRenderImage(RenderTexture src, RenderTexture dst){
         Graphics.Blit(src, dst, mat);
     }   
 }

The behaviour I get (using Unity 5.2.3) from this setup is as follows - the image effect shows the stencil buffer gets created correctly on the first frame after it is activated, but then doesn't get updated again, no matter how much I change the camera or other objects in the scene? (note that I've put [ExecuteInEditMode] on the image effect script, but the exact same problem happens whether I'm in Play mode or Scene mode). (I've also tried placing a full-screen quad at the back of the scene that fills the stencil buffer with "0" just to see if that made a difference but it did not) alt text Disabling and re-enabling the component or the camera itself updates the stencil buffer again. But then, weirdly, resizing the game view window appears to completely clear the buffer - making the screen all black.

Does anyone have any insights or can explain what I'm doing wrong?

stencil.jpg (165.2 kB)
Comment
Add comment · Show 3
10 |3000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image tanoshimi · Oct 13, 2016 at 08:50 PM 0
Share

I will mention that I've already read and tried suggestions in http://answers.unity3d.com/questions/621279/using-the-stencil-buffer-in-a-post-process.html , but I don't think they apply, since it's obvious my stencil buffer is getting written to at least once.

avatar image Bunny83 tanoshimi · Oct 14, 2016 at 11:40 AM 0
Share

Hmm, have you read about the deferred rendering pass on the stencil page? Since Image effects are applied after all rendering has finished you might have some interference with Unity's lighting pass(es). Though you don't seem to specify a renderqueue nor do we know what lighting mode you're using.

I only have played around a little bit with stencil operations in Unity so i've never come across such a problem. I've once implemented stencil shadows (Carmack's reverse) but not in Unity but in plain DX in a C++ application (as an assignment back then ^^).

avatar image tanoshimi Bunny83 · Oct 14, 2016 at 01:02 PM 0
Share

Thanks for the reply @Bunny83 - sorry should have mentioned that I'm using Forward rendering so, to my knowledge, the buffer should not be being interfered with by any lighting calculations.

As I understand, the stencil buffer is stored using 8 bits appended to the depth texture. So I've tried setting Camera.depthTexture$$anonymous$$ode = DepthTexture$$anonymous$$ode.depth to ensure that the depth buffer was being written, and fiddled with 32 bit depth texture on/off, but it made no difference.

0 Replies

· Add your reply
  • Sort: 

Your answer

Hint: You can notify a user about this post by typing @username

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Follow this Question

Answers Answers and Comments

58 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

Stencil Buffers With SpriteRenderer 1 Answer

Stencil Shaders with depth testing 0 Answers

Help with Stencil shaders please 0 Answers

Modifying an image effect component over time. 0 Answers

Unity 5 Deferred Stencil Buffer Usage 1 Answer


Enterprise
Social Q&A

Social
Subscribe on YouTube social-youtube Follow on LinkedIn social-linkedin Follow on Twitter social-twitter Follow on Facebook social-facebook Follow on Instagram social-instagram

Footer

  • Purchase
    • Products
    • Subscription
    • Asset Store
    • Unity Gear
    • Resellers
  • Education
    • Students
    • Educators
    • Certification
    • Learn
    • Center of Excellence
  • Download
    • Unity
    • Beta Program
  • Unity Labs
    • Labs
    • Publications
  • Resources
    • Learn platform
    • Community
    • Documentation
    • Unity QA
    • FAQ
    • Services Status
    • Connect
  • About Unity
    • About Us
    • Blog
    • Events
    • Careers
    • Contact
    • Press
    • Partners
    • Affiliates
    • Security
Copyright © 2020 Unity Technologies
  • Legal
  • Privacy Policy
  • Cookies
  • Do Not Sell My Personal Information
  • Cookies Settings
"Unity", Unity logos, and other Unity trademarks are trademarks or registered trademarks of Unity Technologies or its affiliates in the U.S. and elsewhere (more info here). Other names or brands are trademarks of their respective owners.
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • Default
  • Help Room
  • META
  • Moderators
  • Explore
  • Topics
  • Questions
  • Users
  • Badges