[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Public WebGL] Re: the commit API on the OffscreenCanvas





On Sun, Jul 8, 2018 at 1:19 PM Jukka Jylänki <jujjyl@gmail.com> wrote:

> In the current semantics it's an error to call commit() from inside a requestAnimationFrame callback on a worker.

What is the rationale behind this restriction? Is the intent that
having an infinite loop running in a web worker from within a rAF() is
troublesome?

If you put an infinite rendering loop inside of a rAF callback, then you don't need to be using rAF at all because you are basically ignoring the frame scheduling behavior provided by rAF.
Since commit() and rAF() each have their own scheduling behaviors you should not use them together.  With rAF, there is an implicit commit at the end of all script tasks that draw something. This is compatible with the behavior you get with regular canvases on the main thread, which means that animation code that was written for running on the main thread can be made to work on a worker with minimal code changes.
 
I have been asking before that there was a context creation option to
WebGL that would remove the current implicit "when I return from event
handler, if there were any glDraw()s to front buffer in that event
handler, that's a swap" presentation behavior. It would be nice to be
able to create a WebGL contexts with an explicit swapping behavior
mode, where returning from any event handler would not swap, but
instead swapping would be done only when explicitly told to via a
.commit() option?

Is there a use case that really needs this? In the infinite loop use case, this does not really matter, right? Since the task never ends, there is no implicit swap.

The only case I can think of is if you have multiple event handlers that each draw part of the frame, and you want to prevent partially rendered frames from being displayed.  But I can't think of a reason why an app might need to be written that way.
 

Current Emscripten multithreading web worker GL support has been
implemented from the perspective that also applications that do not
drive their own infinite main loops can still call .commit() to
manually tell when they are finished producing a frame. This feature
would be super nice because it decouples the overall program flow
structure from GL buffers presentation logic, and this decoupling
provides a great deal of flexibility.

Another feature about .commit() is that I would like to see vsync
control either on commit(), or via some other companion function, that
would allow code to control whether .commit() would be
vsyncless/nonblocking, or with vsync swap interval of 1,2,3,etc. And
an API that would allow querying what vsync rate a vsync swap
interval==1 means (in the context of current canvas display or the
context). The rationale for this feature is:
 a. If one knows that one has presentation locked to a particular
vsync rate, one can drive game/animation logic, physics and the rest
at fixed timesteps. Using performance.now() or histogram based
guessing based on rAF() firing rates is troublesome, especially if
under existing GPU load, and will just result in microstuttering.
Being able to read context.vsyncRate or canvas.vsyncRate or something
similar will allow animation to reach microstuttering free timing for
animation.
 b. Displays with >60Hz refresh rates are becoming more common, and
sometimes content wants to constrain to run at known 30fps, or 60fps
(e.g. known 30fps source video animation on a WebGL texture might make
it illogical for the app to render at 60Hz or 120Hz of the source
display). It would be nice to deal with this via a context.commit(1);
or context.commit(2); kind of API to specify the desired swap
interval. (or a separate function would be fine as well). Having a
context.vsyncRate info field would allow computing what the
appropriate swapInterval would be desirable, 1/2th, 1/3th, 1/4th or so
on.
 c. Rendering without vsync enabled at context.commit(0); would be
great for sites that do performance benchmarking or competitive
gaming. This would be fine to be constrained for fullscreen apps for
example. Even if presenting did need to wait for vsync, a
"context.commit(0);" API could be a way to tell "please present when
possible, but don't block on my present API call now".

 Similar ideas have been proposed as extensions for rAF as well. I think this should be in the next feature iteration.  Basically we should add an optional dictionary argument to both rAF and commit() that exposes advanced animation timing options.


In any case, blocking .commit() should never be a full on "glFinish()
right here and now" kind of API, but something that queues new present
work to the swap chain, and immediately returns if there is a free
render target still available on the swap chain (triple buffering).

Agreed.  The current spec is not so explicit on how the throttling should happen because different browsers/OSes use different graphics pipeline models, but basically commit() should only block when the pipe is full. This is similar to how current rAF implementations deliberately skip frames when the GPU can't keep up.
 
Otherwise there will be CPU-GPU pipeline bubbles. In other words,
.commit() would only block when producer is running too far ahead and
has run out of free swap chain buffers to start producing the next
frame to, that it should wait for one to finish presenting to be freed
up for reuse.


2018-07-07 0:32 GMT+03:00 Ken Russell <kbr@google.com>:
> What is known is that some way of committing frames from a spin-loop worker
> is required in the spec, in order to support multithreaded rendering from
> WebAssembly applications. commit() has been tested in small standalone test
> cases. Several groups are collaborating to make multithreaded rendering work
> in a real-world WebAssembly application.
>
> It's a fair point that this should be made to fully work before shipping it
> so we will plan to put commit() back behind a flag in Chrome for the time
> being.
>
> -Ken
>
>
> On Thu, Jul 5, 2018 at 6:29 PM Gregg Tavares <khronos@greggman.com> wrote:
>>
>>
>>
>> 2018年7月6日(金) 5:16 Ken Russell <kbr@google.com>:
>>>
>>> On Wed, Jul 4, 2018 at 11:33 PM Gregg Tavares <khronos@greggman.com>
>>> wrote:
>>>>
>>>> I'm not sure where to bring this up but I've been trying for a couple of
>>>> weeks in other places and getting zero feedback sooo I am hoping you guys in
>>>> charge of things will take a few minutes and read this and take some time to
>>>> thoughtfully respond.
>>>>
>>>> It's possible I don't understand how OffscreenCanvas is supposed to
>>>> work. I've read the spec and written several tests and a couple of short
>>>> examples and this is my understanding.
>>>>
>>>> There are basically 2 ways to use it. They are documented at MDN.
>>>>
>>>> https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas
>>>>
>>>> One is listed as  "Synchronous display of frames produced by an
>>>> OffscreenCanvas". It involves using "offscreenCanvas.transferToImageBitmap"
>>>> inside the worker, transferring that bitmap back to the main thread, and
>>>> calling bitmapContext.transferImageBitmap. This API makes sense to me. If
>>>> you want to synchronize DOM updates with WebGL updates then you need to make
>>>> sure both get updated at the same time. Like say you have an HTML label over
>>>> a moving 3D object.
>>>>
>>>> The other is listed as "Asynchronous display of frames produced by an
>>>> OffscreenCanvas". In that case you just call `gl.commit` inside the worker
>>>> and the canvas back on the page will be updated. This is arguably the more
>>>> common use case. The majority of WebGL and three.js apps etc would use this
>>>> method. The example on MDN shows sending a message to the worker each time
>>>> you want it to render. Testing that on Chrome seems to work but it currently
>>>> has a significant performance penalty.
>>>>
>>>> Recently 2 more things were added. One is that requestAnimationFrame was
>>>> added to workers. The other is the commit as been changed to be a
>>>> synchronous function. The worker freezes until the frame has been displayed.
>>>
>>>
>>> Hi Gregg,
>>>
>>> requestAnimationFrame on workers was added as a result of feedback from
>>> the W3C TAG. It provides a way to animate and implicitly commit frames in
>>> the same way as with HTMLCanvasElement on the main thread. It replaces the
>>> use of setTimeout() on web workers for animating OffscreenCanvases, and
>>> provides a unified mechanism to allow VR headsets to animate at higher
>>> framerates than typical monitors.
>>>
>>>
>>>> It's these last 2 things I don't understand.
>>>>
>>>> First: given that rAF is now available in workers I would think this is
>>>> valid code
>>>>
>>>>       // in worker
>>>>       function loop() {
>>>>           render();
>>>>           requestAnimationFrame(loop);
>>>>           gl.commit();
>>>>       }
>>>>       loop();
>>>>
>>>>
>>>>        {
>>>>           // get messages related to say camera position or
>>>>           // window size or mouse position etc to affect rendering
>>>>       };
>>>>
>>>> Unfortunately testing it out in Chrome this doesn't work. The
>>>> `onmessage` callback is never called regardless of how many messages are
>>>> sent. I filed a bug. Was told "WONTFIX: working as intended"
>>>
>>>
>>> In the current semantics it's an error to call commit() from inside a
>>> requestAnimationFrame callback on a worker. The spec and implementations
>>> should be changed to throw an exception from commit() in this case.
>>>
>>> I updated your samples in your Chromium bug report
>>> http://crbug.com/859275 to remove the call to commit() from within the rAF
>>> callback and they work very well. No flickering, and work exactly as you
>>> intended. Also replied to your same questions on
>>> https://github.com/w3ctag/design-reviews/issues/141 .
>>>
>>>
>>>>
>>>> Really? Is that really the intent of the spec? Apple? Mozilla?
>>>> Microsoft? Do you agree that the code above is not a supported use case and
>>>> is working as intended?
>>>>
>>>> Second: other events and callbacks don't work
>>>>
>>>>       // in worker
>>>>       fetch('someimage.png', {mode:'cors'}).then(function(response) {
>>>>          return response.blob();
>>>>       }).then(function(blob) {
>>>>          return createImageBitmap(response.blob());
>>>>       }).then(function(bitmap) {
>>>>          gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA,
>>>> gl.UNSIGNED_BYTE, bitmap);
>>>>       });
>>>>
>>>>       function loop() {
>>>>           render();
>>>>           requestAnimationFrame(loop);
>>>>           gl.commit();
>>>>       }
>>>>       loop();
>>>>
>>>> This also does not work. The fetch response never comes. My guess is
>>>> this is because in Chrome commit blocks and rAF event gets pushed to the top
>>>> of the event queue so no other events ever get processed. The spec has
>>>> nothing to say about this. Is this supposed to work? It seems like a valid
>>>> use case. Note that switching the end of loop to
>>>>
>>>>           gl.commit();
>>>>           requestAnimationFrame(loop);
>>>>
>>>> also does not work.
>>>>
>>>> Is that correct that it should not work? I guess I don't really
>>>> understand the point of having rAF in worker if these use cases are not
>>>> supposed to work. Are they? If they are not supposed to work can someone
>>>> please explain rAF's use case in a worker?
>>>
>>>
>>> rAF in a worker replaces the use of commit(). Another alternative to
>>> animating in a worker would be setTimeout(), but now that rAF is present in
>>> workers, it's the best alternative.
>>>
>>>> Third, according to various comments around the specs one use case is a
>>>> spin loop on gl.commit for webassembly ports. Effectively this is supposed
>>>> to work
>>>>
>>>>       while(true) {
>>>>          render();
>>>>          gl.commit();
>>>>       }
>>>>
>>>> But I don't understand how this is useful given that no events come in
>>>> if you do that. You can't communicate with the worker. The worker can't load
>>>> files or call fetch or get a websocket message or receive input passed in
>>>> from the main thread or do anything except render.
>>>>
>>>> Maybe people are thinking SharedArrayBuffers are a way to pass in data
>>>> to such a loop but really? How would you pass in an image? As it is you'd
>>>> have write your own decoder since you can't get the raw data losslessly out
>>>> of an image from any web APIs and you can't transfer images into the worker
>>>> (since it's not listening for messages) then you'd need to some how parse
>>>> the image yourself and copy it into a sharedarraybuffer. That would a very
>>>> slow jank inducing process in the main thread so now it seems like the spec
>>>> is saying to use a gl.commit spin loop you need 2 workers, one for
>>>> rendering, one for loading images and other things and then you need 1 or
>>>> more SharedArrayBuffers and you have to implement a bunch of synchronization
>>>> stuff just so you can use WebGL in a worker using this pattern mentioned in
>>>> the spec?
>>>>
>>>> Is that really the intent? Is there something I'm missing? This seems
>>>> like a platform breaking API. Use it and the entire rest of the platform
>>>> becomes unusable without major amounts of code.
>>>>
>>>> If I'm wrong I'm happy to be corrected.
>>>
>>>
>>> commit() is mainly intended to support compiling multithreaded programs
>>> to WebAssembly. The C language's threading model is that threads start up
>>> from a start function and only return from it when the thread exits. We are
>>> trying to get real use cases working which transfer all data in to these
>>> rendering threads via the C heap from other threads. commit() and its
>>> blocking behavior are required in order to reach parity with how native
>>> platforms work in this scenario.
>>>
>>
>> Supporting porting native C app seems like a huge rabbit whole. Are there
>> going to be sync image loading APIs next? Blocking `select` sockets? Reading
>> the clipboard without an event? This is making commit be a huge gate. The
>> moment you use it you throw away the entire rest of the platform. Really?
>> It's hard to believe that's being signed off on.
>>
>> If commit's only use case is native apps it seems like it should not ship
>> and should stay behind a flag until these other issues are worked out. I
>> pointed out several above, like the example that there is no way to use the
>> browser's native image loading with commit even through sharedarraybuffers.
>> It seems very premature to ship such an api (commit) without actually
>> knowing how those issues will be resolved. Tests can be made behind a flag.
>>
>> Are there tests and ports running now with this feature behind a flag that
>> show all these issues can be solved in reasonable ways?
>>
>>>
>>>
>>>> Four: Non front tabs: rAF is currently not delivered if the page is not
>>>> the front tab which is great but rAF is an event so even when rAF stops
>>>> firing because the page is not on the front tab other events still arrive
>>>> (fetch, onmessage, XHR, websockets, etc...) This means even though your page
>>>> doesn't get a rAF callback it can still process incoming data (like your
>>>> chat app's messages).
>>>>
>>>> How is that supposed to work with `gl.commit` loops? It's not the front
>>>> tab so you want to block the commit so the worker doesn't spin and waste
>>>> time. If the worker locks then that seems to have implications for all other
>>>> associated workers and the main thread. If you're using Atomics to sync up
>>>> things suddenly they'll fail indefinitely even more complicating all the
>>>> code you have to write to use this feature.
>>>
>>>
>>> I think that ideally commit() would block until the tab comes back to the
>>> foreground, to minimize CPU usage. However, if that turns out to be
>>> suboptimal for some use cases, we could consider throttling commit(), to
>>> essentially block for some time period and then return control to the
>>> worker.
>>
>>
>> Shouldn't this be figured out before shipping?
>>
>>>
>>>
>>> -Ken
>>>
>>>
>>>> Chrome has already committed to shipping the API. The code as been
>>>> committed so if nothing changes it will ship automatically in a few weeks
>>>> with all the issues mentioned above not behind a flag but live so it seems
>>>> important to understand how to use this and if all these issues were
>>>> considered and what their solutions are.
>>>>
>>>>
>

-----------------------------------------------------------
You are currently subscribed to public_webgl@khronos.org.
To unsubscribe, send an email to majordomo@khronos.org with
the following command in the body of your email:
unsubscribe public_webgl
-----------------------------------------------------------