koobas.hobune.stream
Get the color of a WebCamTexture at the click position
I need to pick the color of a WebCamTexture with a mouse click at this position and render the color to a cube. My scene consists of a quad with the script below and a cube. The quad shows the video content. The cube should get the color of the position where the quad was clicked.
However the cube always gets the color of lower left pixel of the WebCamTexture, no matter at which position of the quad I click to. The RaycastHit seems to output the right coordinates, but it seems to me the GetPixel method doesn´t use the right coordinates. I have seen multiple forum posts but none of them helps me further.
Can you please lift me up? Thanks in advance.
using UnityEngine;
using System.Collections;
public class webcam : MonoBehaviour {
public WebCamTexture wct;
public GameObject cube;
void Start () {
wct = new WebCamTexture ();
GetComponent<Renderer> ().material.mainTexture = wct;
wct.Play ();
}
void Update () {
if (Input.GetMouseButtonDown(0)) {
RaycastHit hit = new RaycastHit ();
Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
if (Physics.Raycast(ray, out hit)){
Debug.Log ("Click position: " + hit.point);
Color color = wct.GetPixel ((int)hit.textureCoord.x, (int)hit.textureCoord.y);
Debug.Log ("Color x/y: " + color);
cube.GetComponent<Renderer> ().material.color = color;
}
}
}
}
Reason of your error: The raycast hit point represents the intersection point in world coordinates.
I'm assuming your quad texture fills the entire screen
1) Your world coordinates have to be converted to screen coordinates. You can use Camera.main.WorldToScreenPoint method to get your screen coordinate.
2) You need to convert the screen coordinates to a ratio. The screen point can't be used as value for the texture coordinates because screen resolutions may be different from the camera texture size. Divide both X and Y by Screen Width and Screen Height to get a 0-1 ratio:
3) Sample texture based using the ratio. You can get the correct pixel by multiplying the texture size by the ratio.
4) Get the pixel using the coordinate.
PS: The reason you're getting the last pixel is because your texture wrap mode is set to clamp, but changing it to repeat wouldn't fix your issue, It would just give you a different result.
PS2: If your camera texture doesn't fill the entire screen (UI element such as RawImage), instead of getting the ratio based on the Screen size, you would have to calculate the ratio based on the UI rect. The formula could vary based on the UI alignment and anchoring.
See my reply... Problem is not entirely solved yet. Can you please explain step 2 more detailed?
Thanks for your detailed answer. I tried to get through it. Step 1 works fine. At step 2 I don´t know exactly what to do.
2) You need to convert the screen coordinates to a ratio.
I added a Unity Package so that you can see what I'm doing. After step 1 I could render colors other than the lower left pixel´s color, but they were not the right colors. After step 2 the result was worse, so I commented it out (script below and in the Unity Package). By the way, my texture does not fill the whole screen. I think it might be easiest for you, me and the other readers on this thread to work with a sample scene, don´t you think so? Thanks in advance.
EDIT:
I tried the same with a movie (another Unity Package attached). Unfortunately, it´s the same effect: I do not get the right color. Someone should explain step 2 to me.
using UnityEngine;
using System.Collections;
public class colorPicker : MonoBehaviour {
public WebCamTexture wct;
public GameObject cube;
Camera cam;
void Start () {
cam = Camera.main;
wct = new WebCamTexture ();
GetComponent<Renderer> ().material.mainTexture = wct;
wct.Play ();
}
void Update () {
if (Input.GetMouseButtonDown(0)) {
RaycastHit hit = new RaycastHit ();
Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
if (Physics.Raycast(ray, out hit)){
Debug.Log ("Click position: " + hit.point);
Vector3 hitpoint = hit.point;
Vector3 screenPos = cam.WorldToScreenPoint(hitpoint);
Color color = wct.GetPixel ((int)screenPos.x, (int)screenPos.y);
// Color color = wct.GetPixel ((int)(screenPos.x/Screen.width), (int)(screenPos.x/Screen.height));
Debug.Log ("Color x/y: " + color);
cube.GetComponent<Renderer> ().material.color = color;
}
}
}
}
Lengthier explanation of 2)
Screen point is a value between 0 and the size of your game window. If run fullscreen on a full hd monitor, it would range between 0 and 1920 in the X axis, and between 0 and 1080 in the Y axis. The Z axis would be zero.
These values are no good for texture lookup, as the camera texture will have the size of the camera resolution, not the monitor resolution. It doesn't matter if your Quad is stretched to fill the entire screen, because the sampling happens at the texture, not the quad. If your camera records at 640x480, sampling a pixel at X:900 and Y:500 would return the wrong pixel, because the texture has the maximum value of 640 for X and 480 for Y.
In order to get the approximate "correct" pixel color, you need to transform the screen point in to a ratio (a percentual) which will represent how far from the origin the pixel position is. Using scenario mentioned above, if the X screen pixel position is at zero, the ratio will be zero. If the X screen pixel position is at 1920, the ratio will be 1. If x position is 960, the ratio will be 0.5. The same rules applies for the Y axis.
Vector2 screenRatio = new Vector2(screenPos.x / Screen.width, screenPos.y / Screen.height);
When you multiply the ratio by the camera texture size (ratio.x to the width, ratio.y to the height), you will get the approximate color of the pixel on that position. Following the example above, a 0 ratio on the X axis, will give you the zero X on the camera texture. The 1 ratio will give you 640 x, and 0.5 ratio will give you 320 x. The same applies for the Y axis.
Color color = wct.GetPixel ( (int)(screenRatio.x * wct.width), (int)(screenRatio.y * wct.height) );
These code snippets not tested.
Thanks for your detailed instructions. Basically it works now. However it only works if the texture has the same pixel ratio as the screen (in other words: The texture must fill the entire screen in order to get the right color). Unfortunately some screens have other ratios. Then you pick the wrong color. I tried to trick around with RectTransforms and colliders, but without effort. The aim is to pick the color of a video playing on a small object in the scene. Do you have the key for the final result?
This ist the script so far:
using UnityEngine;
using System.Collections;
public class WebcamColorPicker : $$anonymous$$onoBehaviour {
public WebCamTexture wct;
public GameObject cube;
Camera cam;
void Start () {
cam = Camera.main;
wct = new WebCamTexture ();
GetComponent<Renderer> ().material.mainTexture = wct;
wct.Play ();
}
void Update () {
if (Input.Get$$anonymous$$ouseButtonDown(0)) {
RaycastHit hit = new RaycastHit ();
Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
if (Physics.Raycast(ray, out hit)){
GameObject hitobj = hit.transform.gameObject;
Vector3 screenPos = cam.WorldToScreenPoint(hit.point);
// map coordinates to display resolution (e. g. 1920 x 1080)
Debug.Log ("Click position relative to the screen size (e. g. 1920 x 1080) " + screenPos);
Vector2 screenRatio = new Vector2(screenPos.x / Screen.width, screenPos.y / Screen.height);
Color color = wct.GetPixel ( (int)(screenRatio.x * wct.width), (int)(screenRatio.y * wct.height) );
Debug.Log ("Color x/y: " + color);
cube.GetComponent<Renderer> ().material.color = color;
}
}
}
}
