koobas.hobune.stream
Timing of WaitForEndOfFrame relative to vertical blank onset?
Hi,
I'm hoping to use Unity for a neuroscience experiment, for which we need very reliable timestamps of event onsets (i.e. when a certain stimulus appears on the monitor or changes). By "reliable" I mean with less than 1 ms jitter, preferably much less.
The standard way this is done is by timing the sending of event codes to the onset of the monitor's vertical blank period. In Unity, there is the WaitForEndOfFrame() class, which is called after all the rest of the processing for the frame has been finished, and I was hoping to use this.
However what I'm wondering is what triggers WaitForEndOfFrame to be active. Is it simply that it happens after all the rest of the processing for the frame has been finished? In this case, it would occur later on frames with more processing, meaning that it would have a large amount of jitter relative to actual frame onset from frame to frame, rendering it fairly useless for us. Alternatively, does it actually get triggered by the vertical blank signal from the monitor? In which case, it is perfect for our needs.
Anything that sheds light on this would be appreciated.
Thanks!
I'm just gonna put it out there, but Unity is a game engine - it is not designed for scientific experiments...
I don't have intimate knowledge of Unity's internal workings (and few here do), but my understanding is that WaitForEndOfFrame is as you describe - it's a queue of events to be processed after all other general game logic in Update() etc. etc. just before the frame is sent to the GPU. I'm not sure how you're defining "jitter", but the length of time between successive WaitForEndOfFrames is certainly variable and would normally be substantially more than 1ms. Or do you mean the time between WaitForEndOfFrame() being called and the screen display itself being updated? That would depend on how much work is being done on the GPU but again would normally be more than 1ms (and, to my knowledge, there would be no reliable way of knowing exactly how long the GPU had taken each frame).
Thanks tanoshimi.
By "jitter" I mean variability in the time taken from the call to WaitForEndOfFrame and the actual beginning of the draw cycle. If WaitForEndOfFrame was synced to the monitor's vertical blank period (either its start or its finish), there should be very little of this variability (1 ms or less).
Ah ok - I understand the question better now. I've never seen anything to suggest that WaitForEndOfFrame is linked to v-sync in any way - it is merely a way to place logic in the game loop that should be applied once all other frame rendering is done. But there may well be a delay after that before the image is pushed to the display. Indeed, depending on exactly how complex the operation is you're intending to perform, you may well be creating such a delay. Either way, I'm not quite sure how this is going to help you, since presumeably your experiment also requires collecting some sort of response by the player to the image on the screen, and that won't be processed until the next frame anyway....
I suspect you're right about WaitForEndOfFrame, unfortunately for me.
The only thing I want to do with this high degree of temporal precision is send a code out over a serial port, to act as a timestamp for our neural recording equipment. This is basically what every piece of neuroscience presentation software does, and it is possible to do it very well. But perhaps not directly within Unity. The reason I asked originally was that other software packages do something that is described in language very similar to WaitForEndOfFrame, but this is synced to the actual physical onset of the monitor's frame.
Everything else, including response collection, can operate with the fuzzy temporal precision that Unity allows. Sadly, looks like I'll have to incorporate tools other than Unity.
I'm also pretty sure WaitForEndOfFrame does not wait for the actual screen refresh. There's no reason not to run it as soon as it can -- it appears to only have been made for what it says -- look at the final render buffer (which would also be after the GPU is finished, right?)
Would Update work? Under Vsynch, it probably runs immediately after the frame displays. $$anonymous$$ight depend on your exact hardware, but seems like it would be easy to test. You can probably find a Com Sci undergrad who'd do it for $25/hour.
On the one hand, maybe the neuroscience journals will mock you for not using a certified vertical blank signal, but Unity is so easy to adjust an experiment overnight that just getting reasonable results with it may be a big deal.
Unity has to ping your hardware at the exact times? I was thinking just count frames and do the math -- but there's no way to insert into your data later?
Update() won't work, no, as it occurs after all the physics calculations are performed. I might be able to do something funky with a FixedUpdate(), which will happen prior to physics if needed, perhaps I could set the interval to 0.1 ms or something crazy...
We are working on a post-hoc way of adding event times into the data (as recorded from photodiodes, which give really nice times) but it would be nice to have accurate timestamps during the experiment, in order to get people to take us seriously if nothing else.
Didn't want to risk overwhel$$anonymous$$g with details, and assumed you weren't running any physics, but using FixedUpdate as "screen hardware is done" should be easy.
Just have it set barely high enough to run (at least) once per frame. The standard 0.02 is plenty. Then a flag you reset in Update for when FU runs twice in a frame (it doesn't really run at the correct intervals -- just one loop each frame to simulate.)
(But still not tested.)
This is the best I can give you. Scroll to the bottom of the link to see the flowchart.
The OP included that link in a mostly-duplicate post to the Forums. But still nice of you to add it. It's sort of buried and it's one of the few pictures which really is worth 1$$anonymous$$ words.
Thanks, I had seen that before, the trouble is it doesn't actually specify the answer any more precisely. I think, though, that the answer is that WaitForEndOfFrame is not synced to the physical frame onset in any way, and simply runs when the rest of the code has finished, making it fairly pointless for my purposes.
I know it doesn't truly answer the question. Unfortunately only Unity themselves would be able to provide a response to that question, and you may have better luck getting that from the Unity Forums ins$$anonymous$$d of Unity Answers, but even that is a toss up sometimes. If you do find the true answer please post it and mark it as the best answer over this one.
As you said, the image does suggest that it simply just runs after the code has finished.
Yeah, I've posted to the forums as well but as yet crickets. I might have to pay for real support or something :).
If you're interested in an alternative hacky way of trying to get ti$$anonymous$$g to do with the start of the frame, rather than the end, see my exchange with Owen-Reynolds above. $$anonymous$$ight be able to use FixedUpdate to get close enough.
Did you ever get more clarity on this question?
I'm trying to do the same thing and have been experimenting with an arduino, a photodiode and an oscilloscope. The results so far are not encouraging. It appears that with vsync enabled there is definitely correlation between the timing of frames in Unity and when the frames get displayed on the monitor. Unity is obviously accessing the vblank signal at some point. As long as things are running smoothly and Unity is keeping up with the frame rate the delay between Unity timing and display timing is consistent. As soon as something disrupts this timing it will take a few frames for Unity to catch up again, but it does recover.
Also, there is some buffering of frames, probably by the video card. Unfortunately the number of buffered frames can change over time causing discreet jumps in the delay time equal to the frame rate. In my case I'm using a Dell S2417DG monitor running at 165Hz. The delay time is usually solid around 16ms but when an additional frame is buffered the delay is ~22ms. Running at 60Hz the delay varies from 50-67ms.
Without accessing the vblank signal directly I currently don't see how to get more accurate timing information.
I also did some checking on the difference between the ti$$anonymous$$g of WaitForEndOfFrame and the start of the next frame in Unity. In general there was a fraction of a millisecond between them (~0.12). Very rarely this would jump to a few milliseconds. $$anonymous$$y tests indicate that WaitForEndOfFrame is the best ti$$anonymous$$g to use for correlation with when the frame actually gets displayed.
Thanks for the information. I've noticed the same issue with regards to Unity ti$$anonymous$$g (as measured by Time.time) being generally fairly close to actual ti$$anonymous$$g as measured by a photodiode (generally within 2 ms). On the occasions when it does go off, Unity is completely untrustworthy for 2-3 frames, and the photodiode ti$$anonymous$$g shows that a frame is "stuck" for 1 or 2 extra frames (i.e. on a 60 hz monitor the frame stays for 33.3 or 50.0 ms).
I believe you when you say that WaitForEndOfFrame is very closely correlated to the ti$$anonymous$$g of the next frame - but I would imagine this isn't true in the cases where you have the ti$$anonymous$$g screwups and "stuck" frames I just described. If so it's not very useful for my purposes. I need to send out codes over UDP with very low latency at the start of frames, as part of a neuroscience experiment. And the whole point of doing this is that even when there are ti$$anonymous$$g SNAFUs the codes still reflect the actual ti$$anonymous$$g of frame changes.
What I've now started doing is sending out codes during the first fixedUpdate in a frame, and have set the fixed update interval to be a suitably low number. So far, the code-to-code latency appears very good, when frames are stuck they show a similar latency difference to the actual display latency as measured with the photodiode.
The only remaining issue as far as I'm concerned is that I'm not fully sure whether the codes are being sent at the start or end of the vblank period. But what I can be sure of is that their offset relative to both the start and the end is constant, otherwise the code-to-code latency would be jumping all over the place.
Does that help? Any suggestions you have are much appreciated.
Thanks for the update!
Have you attempted to measure the delay between frame ti$$anonymous$$g in Unity and when that frame appears on the monitor? I was surprised to find that there are 2-3 buffered frames (presumably in the video card). This, to me, is key because Unity appears to be synchronizing with vblank but it's synchronizing with the vblank of the frame it produced 2-3 Unity updates prior.
In a frame where Unity's update takes longer than the frame rate of the monitor would allow (not precisely true but for illustration) the frame gets displayed for an extra vblank (or more). You are right that in this case WaitForEndOfFrame becomes inaccurate. Unity runs subsequent frame updates as fast as possible until it can recover and re-sync so these frames are also inaccurate.
I don't follow how using fixed update helps in any way. Doesn't fixed update run independently of update and it's update that is synchronized to the vblank?
As for measuring the delay between frame ti$$anonymous$$g in Unity and the monitor ti$$anonymous$$g, no. I haven't really thought of a clever way of doing that yet. Any ideas are appreciated. The issue is I get time measured in seconds since the start of the game (from Time.time) but this isn't easily translatable into system time, which might give me some way of syncing with, e.g., a signal sent from the arduino. And I'm not sure about the accuracy of system timestamps in unity, I remember reading somewhere that they can be off by 10ms or more.
As for FixedUpdate, I have better news there. if you look at the execution order of functions in Unity, you'll see that the first thing that happens on a given frame is the FixedUpdate cycle. So it can be exploited to do stuff that is time-locked to the onset of the frame. The only caveat is that "time-locked to" here means "has a constant latency with respect to", not "happens exactly at the start of". That is, I assume it happens at the end of the vblank, but for all I know it could be the start, or god knows what. But it has a constant latency (this is easy to check with your photodiode, because the times between codes are always going to match the latencies recorded by your photodiode).
This is a little complicated to get your head around, but basically FixedUpdate is there to fake a constant time-step. So lets say you set the fixed update interval to 5ms. The idea is that this simulates events happening every 5 ms. But your frame on a 60hz monitor is 16.7ms. Since this is significantly longer than 5ms, it's a waste of time to actually do all the work associated with generating the full frame every 5 ms. Plus, since Unity doesn't work on anything even close to a real-time schedule, it's never going to be able to actually ensure anything happens every 5 ms (and indeed, this is irrelevant because the only update that actually matters is the 16.7ms one).
So... what actually happens is that unity calculates how many fixed updates it would run in the current frame if these updates were happening at exactly the fixed update interval. In the case of a 60hz monitor with a 5ms fixed update, on frame 1 this would be 3 fixed updates. And then it just runs 3 fixed updates, one right after the other, so assu$$anonymous$$g you're not doing anything crazy the actual fixed update latency within a given frame is probably on the order of 0.01ms or something like that. But it only does it 3 times in that first frame. Then it does the rest of the regular update cycle and everything else. Then it waits for the next frame, and calculates how many fixed updates it should be running during this frame. But this includes the 1.7ms that was left over at the end of the previous frame! So ideally, if the monitor and unity are running in perfect sync, in our example there should be 3 fixedupdates on frame 1, 3 more on frame 2, and then 4 on frame 3.
will continue, running out of space.
WaitForEndOfFrame happens after everything in the frame has been completed: cpu-side camera rendering; animators; post-late-update stuff; your scripts; even the editor overhead. Below is a profiler timeline showing location of coroutines that yield for a new WaitForEndOfFrame. 
