Graphics.Blit not working with my projective texture mapping shader
I am trying to make a simple painting application using projective texture mapping and Graphics.Blit, but it does not work correctly. I am not sure at all what to do. Please help me...
For projection, I made my own script and shader, instead of using Projector component because I want to customize it later. The script is very simple, just passing model, view and projection matrix to shader. The shader is simple as well (just simple projection texture mapping) and I posted it below. I checked if they work correctly, and they do. Next, because I want to reflect the result of the projection to the texture, I replace the main texture with a RenderTexture at first, then call Graphics.Blit every frame. But the after Graphics.Blit, the result texture appears improperly. Can anyone explain why it is not working and how to fix it?
I made a simple script to check everything as below. To try the script, just attach it to a GameObject which the brush texture would be projected on (not that the GameObject is projecting) and set the material with the shader, and then set the brush texture. Also make an empty GameObject which would be a projector and set is as projecor. The texture will be projected towards the object's transfom z. You'll see the projection is working but if you choose "Bake" from the inspector, which uses Graphic.Blit, the projection would be baked at a wrong position
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEditor;
public class ProjectionTest : MonoBehaviour {
[SerializeField] GameObject projector;
[SerializeField] Color brushColor;
[SerializeField] float fov = 20, aspect = 1, zNear = 0.01f, zFar = 1;
Material mat;
int MVPMatPropertyId, brushColorId;
RenderTexture renderTexture;
private void Awake() {
mat = GetComponent<Renderer>()?.material;
MVPMatPropertyId = Shader.PropertyToID("MVPMatForProjection");
brushColorId = Shader.PropertyToID("_BrushColor");
InitCanvas();
}
void InitCanvas() {
var mainTex = mat.mainTexture;
// generate RenderTexture
renderTexture = new RenderTexture(mainTex.width, mainTex.height, 0, RenderTextureFormat.ARGB32, RenderTextureReadWrite.Default);
// copy the main texture to the RenderTexture
Graphics.Blit(mainTex, renderTexture);
// replace the main texture on the material with the RenderTexture
mat.mainTexture = renderTexture;
}
void SetMatProperties() {
if (mat && projector) {
Matrix4x4 modelMat = transform.localToWorldMatrix;
Matrix4x4 projMat = GL.GetGPUProjectionMatrix(Matrix4x4.Perspective(fov, aspect, zNear, zFar), true);
// camera space matches OpenGL convention: camera's forward is the negative Z axis, so negate all of the 3rd row values of the matrix
// https://docs.unity3d.com/ja/current/ScriptReference/Camera-worldToCameraMatrix.html
Matrix4x4 viewMat = Matrix4x4.TRS(projector.transform.position, projector.transform.rotation, projector.transform.lossyScale).inverse;
viewMat.m20 *= -1f;
viewMat.m21 *= -1f;
viewMat.m22 *= -1f;
viewMat.m23 *= -1f;
Matrix4x4 mvpMat = projMat * viewMat * modelMat; // this is the same as UNITY_MATRIX_MVP in shader writing
mat.SetMatrix(MVPMatPropertyId, mvpMat);
mat.SetColor(brushColorId, brushColor);
}
}
// Update is called once per frame
void Update() {
SetMatProperties();
}
[ContextMenu("Bake")]
void Bake() {
Debug.Log("paint");
var renderTextureBuffer = RenderTexture.GetTemporary(renderTexture.width, renderTexture.height); // buffer
Debug.Log("blit");
Graphics.Blit(renderTexture, renderTextureBuffer, mat); // first, copy renderTexture, which the painterMat will be applied to, to renderTextureBuffer because the texture can't be changed directly (and apply the material after copied
Graphics.Blit(renderTextureBuffer, renderTexture); // then, copy the buffer to renderTexture
}
}
shade code
Shader "Custom/Painter/ProjectionPaint"
{
Properties
{
_MainTex("Main Texture", 2D) = "white" {}
_BrushColor("Brush Color", Color) = (1, 1, 1, 1)
_ProjTex("Projection Texture", 2D) = "white" {}
}
SubShader
{
Tags { "RenderType" = "Opaque" }
LOD 100
Pass
{
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#include "UnityCG.cginc"
struct appdata
{
float4 vertex : POSITION;
float2 uv : TEXCOORD0;
};
struct v2f
{
float2 uv : TEXCOORD0;
float4 vertex : SV_POSITION;
float4 projUV : TEXCOORD1;
};
sampler2D _MainTex;
sampler2D _ProjTex;
float4 _MainTex_ST;
float4 _BrushColor;
uniform float4x4 MVPMatForProjection;
v2f vert(appdata v)
{
v2f o;
o.vertex = UnityObjectToClipPos(v.vertex);
o.projUV = ComputeGrabScreenPos(mul(MVPMatForProjection, v.vertex));
o.uv = TRANSFORM_TEX(v.uv, _MainTex);
return o;
}
fixed4 frag(v2f i) : SV_Target
{
fixed4 projColor = fixed4(0, 0, 0, 0);
if (i.projUV.w > 0.0) {
// projection to screen space
i.projUV.x /= i.projUV.w;
i.projUV.y /= i.projUV.w;
if (i.projUV.x >= 0 && i.projUV.x <= 1 && i.projUV.y >= 0 && i.projUV.y <= 1) {
projColor = tex2D(_ProjTex, i.projUV);
}
}
fixed4 mainColor = tex2D(_MainTex, i.uv);
// if _BrushColor.a = 0 or projColor.a = 0, then mainColor. if both 1, then _BrushColor.
return mainColor * (1 - _BrushColor.a * projColor.a) + _BrushColor * _BrushColor.a * projColor.a;
}
ENDCG
}
}
}
a little update:Baking(Using Blit) after moving the projector object doesn't change the result much. Always projected around texture UV(0, 1). Changing the rotation affects a bit, but not much either. Changing the fov affects on the size of the projected texture a lot as expected. So, This shows that values(matrices) are passed to the shader but the calculation of Graphics.Blit is different from normal rendering?
Answer by torano · Mar 09, 2019 at 01:01 PM
I found Graphics.Blit doesn't work with a shader that uses vertices for calculations. Inside the function, something similar to this is executed.
static public void Blit(RenderTexture source, RenderTexture destination, Material material) {
// Set new rendertexture as active and feed the source texture into the material
RenderTexture.active = destination;
material.SetTexture("_MainTex", source);
// Low-Level Graphics Library calls
GL.PushMatrix(); // Calculate MVP Matrix and push it to the GL stack
GL.LoadOrtho(); // Set up Ortho-Perspective Transform
material.SetPass(0); // start the first rendering pass
GL.Begin(GL.QUADS);
GL.MultiTexCoord2(0, 0.0f, 0.0f); // prepare input struct (Texcoord0 (UV's)) for this vertex
GL.Vertex3(0.0f, 0.0f, 0.0f); // Finalize and submit this vertex for rendering (bottom left)
GL.MultiTexCoord2(0, 0.0f, 1.0f); // prepare input struct (Texcoord0 (UV's)) for this vertex
GL.Vertex3(0.0f, 1.0f, 0.0f); // Finalize and submit this vertex for rendering (top left)
GL.MultiTexCoord2(0, 1.0f, 1.0f); // prepare input struct (Texcoord0 (UV's)) for this vertex
GL.Vertex3(1.0f, 1.0f, 0.0f); // Finalize and submit this vertex for rendering (top right)
GL.MultiTexCoord2(0, 1.0f, 0.0f); // prepare input struct (Texcoord0 (UV's)) for this vertex
GL.Vertex3(1.0f, 0.0f, 0.0f); // Finalize and submit this vertex for rendering (bottom right)
GL.End();
GL.PopMatrix(); // Pop the matrices off the stack
}
I am not really sure about GL class, but I guess vertex positions of a quad are set by GL.Vertex3 and they go directly to shader. Thus, projUV in my projective shader was not set correctly.
But I got a solution from the other website, and according to it, I could use Graphics.DrawMeshNow or CommandBuffer.DrawRenderer instead. They seem to work to save the result of the projection.