[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Public WebGL] WEBGL_dynamic_texture extension proposal




On Jul 6, 2012, at 2:27 PM, Florian Bösch <pyalot@gmail.com> wrote:

On Fri, Jul 6, 2012 at 11:17 PM, Chris Marrin <cmarrin@apple.com> wrote:
I think our goal should be to allow full frame rate playback of perfectly synced and paced video in a 3D scene. But to do that we need to do better than just letting the video decoder give us whatever frame happens to be ready. The renderer really needs full awareness of media timing, which needs to be communicated in both directions between the media provider and the renderer.
The <video> container should know which frame (time) is desired pretty precisely doesn't it (after all we also control video trough it)? And the <video> container would have some of that mechanism you mention to show the frame it wants. And wouldn't that frame be held (in normal operation) in video memory? So I think the entire mechanism and the parameters to the mechanism is there, it seems just a matter of exposing a "sync for current time now" to JS.

The video decoder has no idea (nor should it) how fast or how consistently a 3D frame is being rendered. So it's possible that the time at which we need to acquire the video frame will be many ms before it actually makes it to the display. We might not even be acquiring the frame the same number of ms in advance in every frame. So we could hit on one side or the other of a new frame being available and that will cause inconsistent video playback.

When we use requestAnimationFrame to time the WebGL rendering, we have the timestamp for the start of that cycle, which will be very consistent. If we pass that to the video acquire function, then we will get a very consistent frame progression, even if our WebGL rendering is not making framerate. 

And think about the future when we will be able to render multiple WebGL frames at a time. In that case we will need to tell the acquire function when we intend to display the given frame or we will get wildly inconsistent video playback. 

Video is timed media. I believe we need to make our requests to it explicitly time based, or we'll have playback quality issues, and we will limit our ability to effectively use future hardware.

-----
~Chris Marrin
cmarrin@apple.com