koobas.hobune.stream
Image Quality Degradation between External Camera Stream and Display in Unity
Hey, guys,
I'm working on the Valkyrie project for NASA (Valkyrie / R5), and there's a few questions I'd like to ask you fine folks that relate to a single issue.
The Valkyrie's being fitted for virtual reality teleoperation, with two cameras outputting their respective feeds through ROS (https://www.ros.org/) as JPEG images at around ~ 5 Hz. We're using the ROS# Unity plugin in order to get the images from the cameras and display them in a virtual reality headset via Unity; I used an approach similar to this one: https://github.com/guiglass/StationaryStereoCamera in order to get two cameras looking at Plane objects which have the camera images rendered onto them via the material's SetTexture function.
The Problem:
The images as displayed on a Unity Plane are significantly degraded in quality from the original image stream from the cameras on the Valkyrie. I know that a texture's quality can be tuned using the import settings, but unlike in #447719, my textures are being generated at runtime, so there's no import setting window.
(^ Bad quality texture from Unity.)
(^ Good quality texture from the cameras themselves.)
Despite resizing the planes to match the aspect ratio of the 1328 x 1048 images, the weird pixelization effect persists.
The Question: I can't find any settings in the scripting API for Texture2D resembling the settings mentioned in [5]. Is there a way to do this sort of thing in another type of image/texture?
If ya'll want any more information about my setup, scene, scripts, or anything, just let me know either in the comments or in a PM. I couldn't include the package because it was too big at ~900 kB.
[2]: https://github.com/guiglass/StationaryStereoCamera
Part of code responsible for generating texture would certainly not be a bad thing.
Also quick question: if you import this good quality image into Unity, does it look as bad as the generated one?
In response to @Casiell, here's the script for generating the texture from an inco$$anonymous$$g array of bytes.
public sealed class ValkyrieCameraSubscriber_3D : UnitySubscriber<$$anonymous$$essageTypes.Sensor.CompressedImage>
{
public $$anonymous$$eshRenderer meshRenderer;
private Texture2D texture2D;
private byte[] imageData;
private bool is$$anonymous$$essageReceived;
protected override void Start()
{
base.Start();
texture2D = new Texture2D(Screen.height, Screen.width);
meshRenderer.material = new $$anonymous$$aterial(Shader.Find("Unlit/Texture"));
}
private void Update()
{
if (this.is$$anonymous$$essageReceived)
{
Process$$anonymous$$essage();
}
}
private void Process$$anonymous$$essage()
{
texture2D.LoadImage(imageData);
texture2D.Apply();
meshRenderer.material.SetTexture("_$$anonymous$$ainTex", texture2D);
is$$anonymous$$essageReceived = false;
}
protected override void Receive$$anonymous$$essage($$anonymous$$essageTypes.Sensor.CompressedImage image)
{
imageData = image.data;
is$$anonymous$$essageReceived = true;
}
}
Receive$$anonymous$$essage is a callback function that is invoked each time a ROS message comes through. Upon receipt of a CompressedImage, the imageData variable is updated with the image bytes. Then, Process$$anonymous$$essage is called, which invokes LoadImage on the Texture2D object to get the image data in there, then it uses Apply() to ensure that it'll display right (the documentation told me this step was necessary). Then we set the plane's texture to this generated texture.
As for the importation of the good quality image, it appears that it retains excellent sharpness if I import a snapshot of the camera feed like a regular texture. 
I just found a way to fix my problem. There are two RenderTextures in my scene that were hooked into the cameras for each eye. The two RenderTextures were only 256x256, so the 1328x1048 camera feed image was being stuffed into a texture of that size. I solved it by making the RenderTextures bigger than the incoming images.
I'll consider this question answered (by myself).
