koobas.hobune.stream
Is it feasible to play 4K video in Unity?
I'm starting work on a 360 VR video player, aimed at mobile devices (iOS/Android Cardboard + GearVR) and I'm evaluating whether it is possible to get video quality from 1080p to 4k in Unity or if I should instead just do the project natively (aka, OpenGL).
I've been trying to find information on the state of video in Unity but I haven't arrived to a clear answer. I'm going to explain the info I'm running on as of now, hopping that someone can add to this or provide an answer or at least show that I am erring on the side of ignorance :)
Assume a video in mp4, H264 at 4K resolution (equirectangular mapping):
OpenGL Approach:
If I were to do this in OpenGL, a (somewhat simplified) workflow would entail decoding each video frame, updating a GL_TEXTURE_2D with the pixel contents, and rendering to a sphere with glCullFace(GL_FRONT) as if it where a regular skybox.
With the above setup, the limiting factors would be:
Does the GPU in my device support 4K textures? what marketing calls 4k is usually 3840 horizontal, and, an iPhone6 for instance, returns 4096 for
glGetIntegerv(GL_MAX_TEXTURE_SIZE,...). So I'd assume the iPhone6 can load textures as big as required for 4K.Fill rate: i.e.: my iPhone6's GPU might be capable of storing 4K Textures, but it might not be fast enough to fill up a 4K frame buffer at an acceptable frame rate. This unfortunately I think the only way to know is by testing. There is also the added overhead of the Cardboard rendering which means 2 passes per frame.
Unity Approach:
Now for a Unity approach, this is what I did:
Empty scene with a sphere and cardboard camera at the origin.
For the sphere, I created a material based on the Unlit shader with one texture and set the cull face to front.
imported the video and assigned it as the material's texture.
My observations from the above test are:
UPDATE: I now realize that my test was using Unity's Move Texture, which is not available on mobile. So if I still pursuit this project in Unity I will anyway have to either do my own native Unity plugin and write the OpenGL code to map the textures to the geometry or purchase an asset like AVPRO
When importing images, Unity also allows texture sizes of 4096 so 4K should be possible.
Importing a video however, takes a ridiculous amount of time. I believe this is because Unity actually uses Ogg Theora and so it takes forever to transcode to Ogg. As a matter of fact, sometimes I was able to import and sometimes Unity would just stall.
When importing a video, I was not given any mmap, filtering or size options as if I was importing an image. So I'm not really sure what Unity is doing to the video. All I get is a quality / compression slider. The end video is still 3840x1920, which was the original resolution.
The rest of the experiment works ok, as in, yes, the video playbacks in the sphere. But the framerate is not great.
So at this point my conclusions are:
Is it possible to do 4K in Unity? Kind of, In the sense that you can import 4K videos and reproduce them without any external assets.
Is it really 4K quality? Probably not. mp4 is lossy and so is ogg, So I assume my video went a second round of lossy compression when it was transcoded to Ogg.
Is it practical? Not sure. So far, video importing takes an impractical amount of time. Also, 4K in general might be too much for a mobile device rending in stereo, regardless of Unity or doing it straight in OpenGL.
As a side note and to complicate things even more, I was thinking of allowing both downloading and streaming video in the app. But from the above it seems that would be out of the question performance wise. Maybe I have to stick to 1080p or even 720p?
It's possible to play 4k videos in Unity, but from personal experience without external assets, like you said, the framerate is not great and it could take hours to import.
So for myself, I use easy movie texture and it's totally different, it's fluid and in the StreamingAssets folders Unity doesn't convert the videos to Ogg, so it's really fast.
I was currently trying out the demo for AVPro, substantially more expensive than Easy $$anonymous$$ovie Texture. So you are saying Easy $$anonymous$$ovie is enough to play 4$$anonymous$$?
Also, yes, in the meantime I discovered that if I import to the strea$$anonymous$$g folder I skip the processing time since the files are left intact, like you said.
And do you think it might be worth exploring me just doing a native plugin ins$$anonymous$$d? Have you tried this approach? The potential would lie on me making sure only the necessary GL commands are issued for just this use case, but it might just not be significantly more performant.
Thanks!
I had no problem with Easy $$anonymous$$ovie Texture (I made Cardboard and GearVR app with it), but I never played 4k videos with AVPro, so I can't compare.
I never really think about doing a native plugin, because I was using easy movie texture and it would take a lot of time for me to do it. The only problem with easy movie texture is that it only work on mobile.
There is no demo for Easy $$anonymous$$ovie Texture right? as in, a watermarked version but that I can still test in the context of my project? I emailed the developer but haven't heard back yet.
About your last comment, what do you mean it won't work on mobile? Easy $$anonymous$$ovie Texture itself or playing 4$$anonymous$$ video thru it or what? Damn.. I just bought the plugin :/
I was talking about the Editor mode, it's free so you can test it, but it only work in editor. Don't worry ^^
Does easy movie texture work for pc releases too? All I find are android examples but I'd like to play a 4k Video for the HTC VIV$$anonymous$$
Nope, since it is basically a bridge into the native Android and iOS respective media frameworks.
Your options for PC would be:
Let Unity do the importing and conversion of the video by NOT placing it in the Strea$$anonymous$$g Assets folder and then playing it back with the $$anonymous$$ovie Texture from Unity.
If you want something that works for all platforms you could try AVPro. I'm pretty sure it supports PC too; it has a demo, so you could try it out.
Alternatively you'd have to make your own plugin, which would basically amount to finding out what API Windows normally uses for video playback (in iOS/OSX you have AVFoundation for instance, which is what E$$anonymous$$T uses for iOS -- I have no idea on Windows) and writing a wrapper for it from whatever language the API natively supports (I'd guess C++) to a plain C bridge header. Then you compile that into a lib and call it from Unity / C# via interop services.
Hi @Sephis i am using Easy-$$anonymous$$ovie-Texture. But even if I play 4k video (for VR project), it seems of less quality. Do you have any idea why is it so? The video is playing in the 4k supported devices, but it doesn't have the quality.
I don't really know, it could be your video's bitrate, for a 4k video, you need an higher bitrate than a 1080p video for example.
In case if anyone will be searching for a solution for desktop platforms in this topic (like I did), there is Demolition Media Hap plugin, which is capable of playing videos up to 8k@60fps!
Great find! Do you by chance know if it works that good in a VR environment?
Hi! Could you please add me on skype, using the same username as here? I think we can figure out something
Can't right now. Will come back to you when I have some spare time!
Hi @saldavonschwartz, did you ever manage to get 4K video textures running at a decent frame rate in Unity?
I'm trying to create a project whereby the user watches a 360 video and then gets presented with two buttons to choose from which then selects the next scene. But it all rests on 4K textures running at a decent frame rate. Any help would be greatly appreciated. Thank you.
hey, I was writing a detailed explanation but it got moved to moderation somehow. I don't even understand this forum...
So I'll give you the short version.
There is definitely some downsampling going on in the whole process from the decoding of the video to the final image you see when you run your Unity project. You feed 4$$anonymous$$ video, but you don't get actual 4$$anonymous$$ quality frames out.
Things you could try are:
Write your own decoder to make sure it does not downsample or otherwise degrade the original video.
$$anonymous$$ake sure you feed high res textures to the material in the sphere you project onto.
tweak Unity's native VR 'scale'. I believe they expose a scale property in the VR class in the C# api. This seems to allow downsampling or oversampling the 2 eyes textures. Of course higher scale comes with a performance penalty since you are increasing fillrate demands in the GPU.
Other than that, and since Unity is closed source so we really have little idea how things are being implemented, the only way you can assure yourself a high quality video player experience is by coding it all yourself from scratch, using OpenGL for instance.
To be honest I am not really sure what the quality issue is. Since Unity is a black box (closed source) there is no straight forward way for me to look at what is happening.
What I can tell you is this:
The basics: To playback a 360 video in VR you need to do the following:
Decode video frames into gpu video memory and keep updating this memory region each frame. This can be done thru Unity's $$anonymous$$ovie Texture mechanism or an external plugin, like Easy $$anonymous$$ovie Texture or AVPro.
Project that texture onto a spherical geometry. You do this by creating a sphere in some modeling software or thru code in Unity, where you negate each normal so that it points inwards, since setting culling to front face will allow you seeing "inside" the sphere but still the image would be "flipped". If you want to try and further optimize things, write a bare bones vertex / fragment shader pair where you literally just do a texture lookup, to avoid any extra computations, lighting, etc in the default shader.
Place the camera in the center of this sphere. Alternatively you can also pass the position of each vertex of the sphere in the shader from point 2 as position = v.postion.xyww, causing z to be infinitely away (you'd basically be creating the equivalent of a skybox that is always infinitely away).
Render that camera. Since we are talking VR, this implies render to texture (RTT). Specifically, rendering to two textures: one for each eye, one per pass. The exception here would be if you are targeting a system with a GTX1080 for instance, where you have support for multiple targets in one pass.
Apply distortion correction and render one final texture with both of the previous textures.
in my case with easy movie texture 4k video having 40 mb does not streming well. it just stop playing after playing some frame. if put this video in streaming asset folder it works well. but i need to play it from stream .. plz help
Yes! I faced the same problem few months ago. Don't know the solution... Any updates?
With Unity 5.6, a new component called VideoPlayer has been introduced. Check it out. It plays 4k videos easily! Currently no Android support though.
