[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Public WebGL] WEBGL_dynamic_texture extension proposal



Below is some feedback from Acorn Pooley originally posted to the working group mailing list.

Regards

    -Mark

------------------------------------------------

Feedback on the WEBGL_dynamic_texture extension:

GENERAL FEEDBACK:

The extenstion is written based on the semantics of EGLImages.  However there
are a number of problems with the EGLImage semantics which EGLStream has
addressed.  I think it would benefit WebGL to adopt the EGLStream semantics
rather than the EGLImage semantics.  The WebGL extension could still be
*implemented* using EGLImages (or even regular 2D textures using glTexImage2D),
with the browser implementing the more EGLStream-like semantics.

An example is the use of a texture that is acquired.  With EGLImage semantics
(and current WEBGL_dynamic_texture semantics) the texture can be modified
asynchronously at any time while it is being used (this results in
implementation dependent behavior which will vary from one implementation to
another).  With EGLStream semantics the acquired image can never be modified
while it is acquired.  This works because the EGLStream images are actually
double (or triple or more) buffered.  The producer is only allowed to modify
images which are not currently acquired.  This is easy to implement in the
browser using EGLImages -- the browser just needs to create 2 or more EGLImages
and implement the double buffering itself.  This is especially critical for
video sources where it makes no sense to allow the decoder to asynchronously
modify the texture that is being used for rendering.  But the EGLStream
multi-buffered semantics would also be much easier to use and less error prone
(and more similar between implementations) when applied to image and canvas
sources.

Another example is the use of a dynamic texture after
dynamicTextureReleaseImage has been called.  EGLStream semantics require that a
TEXTURE_EXTERNAL texture that has no image acuired be treated as incomplete --
sampling it always returns black (0,0,0,1) pixels.  The semantics of EGLImage
which are in the current extension wording are basically "undefined results"
which will make life difficult for developers and make content appear
differently on different implementations.

I have noted below in "SPECIFIC FEEDBACK" the details.


Another area that much thought has gone into is how TEXTURE_EXTERNAL should
work when the source is a video which is playing at a different rate than the
WebGL rendering loop.  EGLStream addresses this by stating that "Acquire" gets
the image which should be displayed next.  If the video is playing back very
quickly relative to the rendering then this may result in frames being skipped
(which is good because the user sees the frame they should be seeing "now"
which is synced to the audio they are hearing "now").  If the video is playing
back more slowly then 2 or more calls to Acquire may return the same frame
(also good since again the user sees what they should be seeing "now").  The
current WEBGL_dynamic_texture extension does not preclude this behavior, but in
fact does not address it at all.  I think it is a good idea to require it to
work this way.  Otherwise there is too much flexibility and different WebGL
implementations may act differently.

(There is also the GL_KHR_stream_fifo extension which allows the app to opt in
to different semantics (never drop frames) but that is probably less useful --
you can decide if you need that option or not.)


SPECIFIC FEEDBACK:

- In EGL_KHR_stream_consumer_gltexture the acquire/release functions work like
  this:

    Before dynamicTextureAcquireImage is called the texture is incomplete -
    this is well defined and acts like a texture with all black pixels (i.e.
    every pixel is 0,0,0,1).

    After dynamicTextureReleaseImage is called the texture is again incomplete.

    Calling dynamicTextureReleaseImage is not required.  A call to
    dynamicTextureAcquireImage implicitly first calls
    dynamicTextureReleaseImage if it has not been called since the last
    acquire.

    While an image is acquired (i.e after a successful call to
    dynamicTextureAcquireImage and until a call to dynamicTextureReleaseImage)
    the source is free to continue producing new frames.  These changes do not
    affect the acquired image.  Basically the producer is modifying a different
    buffer than the one that the texture is using.  Having an image acquired
    has no affect on the producer.

This is better defined and less error prone than the current
WEBGL_dynamic_texture wording.  I think it would benefit WebGL to work a little
more like EGL_KHR_stream_consumer_gltexture.


- In section 5.13.8.1 it says
        Dynamic textures are texture objects whose images change outside the
        control of the WebGL application and asynchronously with its rendering.
    I think this is a problem.  The textures can be *acquired* from objects
    whose images chage outside and asynchronously.  But the image displayed by
    the texture itself should only updated when the dynamicTextureAcquireImage()
    function is called.  I think this is very important.  Perhaps change
    the opening sentences to
        Dynamic textures are texture objects whose images can be acquired from
        sources outside the WebGL application.  The image source changes
        outside the control of the WebGL application and asynchronously with
        its rendering, but acquisition of the image into the WebGL texture is
        synchronous.


- In section 5.13.8.1 Dynamic textures: the description of the
  dynamicTextureAcquireImage() function states that it locks the image so it
  cannot change.  Really the operation that we want to occur is that the image
  gets latched.  The source of the images can continue to create new frames of
  image data and is not really affected by the acquire call.
  (Implementation note: when implemented with EGLImages this is accomplished by
  the browser creating 2 EGLImages and double buffering.  When implemented as
  glTexImage2D it is even easier since each acquire corresponds to a call to
  glTex[Sub]Image2D.)

- In section 5.13.8.1 Dynamic textures: the description of
  dynamicTextureReleaseImage says that the image may be modified asynchronously
  while being sampled.  I think this is a bad idea.  It will lead to
  differences in behavior between different implementations.  The
    https://cvs.khronos.org/svn/repos/registry/trunk/public/egl/extensions/KHR/EGL_KHR_stream_consumer_gltexture.txt
  extension solves this by making the texture incomplete at times that it is
  not acquired.  In other words, when no acquired sampling the texture always
  returns well defined black pixels.  I think this is a better choice of
  semantics.


- In section 5.13.8.1 Dynamic textures: It says
        If a texture object to which a dynamic source is bound is bound to a
        texture target other than TEXTURE_EXTERNAL the dynamic source will be
        ignored. Data will be sampled from the texture object's regular data
        store.
    I think it is a mistake to allow the same texture to be bound at one time
    to a TEXTURE_EXTERNAL target and at another time to a different target.  In
    GLES2 this is an error (INVALID_OPERATION).  Once a texture is bound to a
    TEXTURE_EXTERNAL target it may never be bound to any other target.  Once a
    texture is bound to a target other than TEXTURE_EXTERNAL it may never
    be bound to the TEXTURE_EXTERNAL target.  (Maybe it is not possible to have
    the same semantics in WebGL, but if it is possible I think this is
    preferable.)




- Note that the  GL_OES_EGL_image_external extension describes how the
  GL_TEXTURE_EXTERNAL_KHR texture target works with EGLImages.  It is the
  GL_NV_EGL_stream_consumer_external extension which describes how
  GL_TEXTURE_EXTERNAL_KHR works with EGLStreams.  It might make more sense to
  mention GL_NV_EGL_stream_consumer_external instead of (or in addition to) the
  GL_OES_EGL_image_external extension.  The two extensions are very similar so
  this may not really matter.

  Actually it is the EGL_KHR_stream_consumer_gltexture which really describes
  how EGLStreams work with TEXTURE_EXTERNAL.  It might be worth referring to
  this extension and/or borrowing some of the language about Acquire/Release
  functions.


Thanks
-Acorn
--