[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Public WebGL] OES_EGL_image_external Extension proposal

Unless I'm mistaken, knowing the time alone isn't going to help that much.

The dynamic_texture extension attempted to solve this problem with setting a presentation time for a frame. I'm not sure how realistic it is to try to time presentation time, so that might have been overly ambitious.

On Fri, Jan 6, 2017 at 8:01 PM, Byungseon Shin <sun.shin@webkit.org> wrote:
Hi WebGL working group members,

We have updated proposal by adding following function : 
- Provides time of frame, texture width and height of HTMLVideoElement's EGLImage.

Please take a look at the pull request:

Kind regards,
Byungseon Shin

On Tue, Jan 3, 2017 at 8:48 AM Byungseon Shin <sun.shin@webkit.org> wrote:
Hi Florian,

We are working on update which will cover dynamic_texture concerns.
I plan to upload new version on github for reviewing by this week.

Happy 2017 year!
Byungseon Shin

On Tue, Jan 3, 2017 at 1:58 AM Florian Bösch <pyalot@gmail.com> wrote:
What is the status of the https://www.khronos.org/registry/webgl/extensions/proposals/OES_EGL_image_external/ extension? Is work on the spec still going on or is this the final version that takes into account dynamic_texture concerns?

On Wed, Oct 12, 2016 at 3:59 AM, Byungseon Shin <sun.shin@webkit.org> wrote:
Hi Ken,

Thanks for the feedback.
Of course, we are willing to devote time and eager to enhance WebGL video rendering performance.
We prefer to use our proposal as a base but we could revise dynamic_texture as well.

Kind regards,
Byungseon Shin

On Wed, Oct 12, 2016 at 8:48 AM Kenneth Russell <kbr@google.com> wrote:
Hi Byungseon,

Thanks for putting together this extension proposal.

In conversations with groups within Google that are trying to do 360 degree video rendering in WebGL, they need more information than your extension would provide; specifically, they would need the exact timestamp of the frame currently being rendered, as well as other per-frame metadata like the current width and height of the texture (for variable bitrate video streams).

Mark Callow's WEBGL_dynamic_texture extension proposal https://www.khronos.org/registry/webgl/extensions/proposals/WEBGL_dynamic_texture/ provides the controls needed. However, per working group discussions, the current extension's a little too complicated. I think if that extension were simplified a little bit that it would provide all of the performance benefits yours offers, and the controls that are known to be necessary.

Do you think you'd be willing to devote some time to either extending your proposal (I can provide specific feedback) or editing down Mark's proposal to handle these use cases? I'd prefer to edit down WEBGL_dynamic_texture, if you're willing.



On Tue, Oct 11, 2016 at 8:27 AM, Byungseon Shin <sun.shin@webkit.org> wrote:
Hi Corentin,

On Tue, Oct 11, 2016 at 11:16 PM Corentin Wallez <cwallez@google.com> wrote:
The problem with using an EGL_image type of extension is that the texture data store becomes shared between the new texture and the object it was created from. This is great when the application controls both sides and can ensure read and writes are probably synchronized, but in the case of video here, one side would write without the application being able to control synchronization.

The standard way to do this type of video / texture binding in EGL is to use EGL_KHR_stream which is a much more complex and might not be an improvement over what developers can do with the current APIs.

As in proposal revision#3 and as Florian already mentioned,
Current proposal support "bind once and update implicitly" concept :  "ext.EGLImageTargetTexture2DOES(ext.TEXTURE_EXTERNAL_OES, videoElement); that the texture is "bound" to the video and no further calls need to be emitted (as would presently be the case)" 

In a that sense, we could abstract the details implementation into MediaPlayer Side like synchronizing issues.

Even EGL_KHR_stream provides more details synchronization issues but still lot's of GPU driver and Video Decoder need to support the latest specification. 
But OES_EGL_image_external extension is already supported by most of the GPU drivers and Video decoders.

And one of the performance bottleneck is to converting EGLImage input to TEXTURE_2D to handled by WebGL application. 
On Tue, Oct 11, 2016 at 6:54 AM, Byungseon Shin <sun.shin@webkit.org> wrote:
Hi Florian, Maksims,

Thanks for the feedback.

I have updated proposal to explain how application works.
Please see the attached updated proposal.

To render an update frame, we need to call ext.EGLImageTargetTexture2DOES to bind newly generated texture buffer from video decoder.

By supporting OES_EGL_image_external extension, application can use direct texture when video driver provides EGLImage like OpenGL ES natvie applications.

Our proposal is just focusing on extending format of texture compatible with OpenGL ES extension and does not conflict with previous proposals.

So, we provides a simplest way to adopt OES_EGL_image_external extension.

By using references implementations of browser, WebGL renders video more than 10 times faster than TEXTURE_2D format with Full HD resolution output. 

Kind regards,
Byungseon Shin

On Tue, Oct 11, 2016 at 4:51 PM Maksims Mihejevs <max@playcanvas.com> wrote:

Worth mentioning that in web there are multiple sources that have independent redraw mechanics, that includes video as well as canvas elements. It might make sense to have single extension for providing direct access to those sources without need to re-upload it.

I have assumptions that internally in browsers, both video and canvas have their buffers, so it might lead to very similar implementation as from internal side as well as from webgl api side.

On 11 Oct 2016 12:12 p.m., "Florian Bösch" <pyalot@gmail.com> wrote:
Am I correct in assuming that once you've called ext.EGLImageTargetTexture2DOES(ext.TEXTURE_EXTERNAL_OES, videoElement); that the texture is "bound" to the video and no further calls need to be emitted (as would presently be the case) for it to stay current?

If that's the case, I think it's a good idea, it takes some work off web application programmers to make sure the video is updated, and it should also in all cases avoid a texture download/reupload.

There are some other extensions that have attempted to deal with this problem.
 Perhaps you could elaborate a little why OES_EGL_image_external would be preferrable to WEBGL_dynamic_texture, and how the issues that EGL_image_external does not tackle (timing, format conversion, other things) would be handled by the web application programmer?