[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Public WebGL] the commit API on the OffscreenCanvas



I'm not sure where to bring this up but I've been trying for a couple of weeks in other places and getting zero feedback sooo I am hoping you guys in charge of things will take a few minutes and read this and take some time to thoughtfully respond.

It's possible I don't understand how OffscreenCanvas is supposed to work. I've read the spec and written several tests and a couple of short examples and this is my understanding.

There are basically 2 ways to use it. They are documented at MDN.

https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas

One is listed as  "Synchronous display of frames produced by an OffscreenCanvas". It involves using "offscreenCanvas.transferToImageBitmap" inside the worker, transferring that bitmap back to the main thread, and calling bitmapContext.transferImageBitmap. This API makes sense to me. If you want to synchronize DOM updates with WebGL updates then you need to make sure both get updated at the same time. Like say you have an HTML label over a moving 3D object.

The other is listed as "Asynchronous display of frames produced by an OffscreenCanvas". In that case you just call `gl.commit` inside the worker and the canvas back on the page will be updated. This is arguably the more common use case. The majority of WebGL and three.js apps etc would use this method. The example on MDN shows sending a message to the worker each time you want it to render. Testing that on Chrome seems to work but it currently has a significant performance penalty.

Recently 2 more things were added. One is that requestAnimationFrame was added to workers. The other is the commit as been changed to be a synchronous function. The worker freezes until the frame has been displayed.

It's these last 2 things I don't understand.

First: given that rAF is now available in workers I would think this is valid code

      // in worker
      function loop() {
          render();
          requestAnimationFrame(loop);
          gl.commit();
      }
      loop();

      {
          // get messages related to say camera position or
          // window size or mouse position etc to affect rendering
      };

Unfortunately testing it out in Chrome this doesn't work. The `onmessage` callback is never called regardless of how many messages are sent. I filed a bug. Was told "WONTFIX: working as intended"

Really? Is that really the intent of the spec? Apple? Mozilla? Microsoft? Do you agree that the code above is not a supported use case and is working as intended?

Second: other events and callbacks don't work

      // in worker
      fetch('someimage.png', {mode:'cors'}).then(function(response) {
         return response.blob();
      }).then(function(blob) {
         return createImageBitmap(response.blob());
      }).then(function(bitmap) {
         gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, bitmap);
      });
      
      function loop() {
          render();
          requestAnimationFrame(loop);
          gl.commit();
      }
      loop();

This also does not work. The fetch response never comes. My guess is this is because in Chrome commit blocks and rAF event gets pushed to the top of the event queue so no other events ever get processed. The spec has nothing to say about this. Is this supposed to work? It seems like a valid use case. Note that switching the end of loop to

          gl.commit();
          requestAnimationFrame(loop);

also does not work.

Is that correct that it should not work? I guess I don't really understand the point of having rAF in worker if these use cases are not supposed to work. Are they? If they are not supposed to work can someone please explain rAF's use case in a worker?

Third, according to various comments around the specs one use case is a spin loop on gl.commit for webassembly ports. Effectively this is supposed to work

      while(true) {
         render();
         gl.commit();
      }

But I don't understand how this is useful given that no events come in if you do that. You can't communicate with the worker. The worker can't load files or call fetch or get a websocket message or receive input passed in from the main thread or do anything except render.

Maybe people are thinking SharedArrayBuffers are a way to pass in data to such a loop but really? How would you pass in an image? As it is you'd have write your own decoder since you can't get the raw data losslessly out of an image from any web APIs and you can't transfer images into the worker (since it's not listening for messages) then you'd need to some how parse the image yourself and copy it into a sharedarraybuffer. That would a very slow jank inducing process in the main thread so now it seems like the spec is saying to use a gl.commit spin loop you need 2 workers, one for rendering, one for loading images and other things and then you need 1 or more SharedArrayBuffers and you have to implement a bunch of synchronization stuff just so you can use WebGL in a worker using this pattern mentioned in the spec?

Is that really the intent? Is there something I'm missing? This seems like a platform breaking API. Use it and the entire rest of the platform becomes unusable without major amounts of code.

If I'm wrong I'm happy to be corrected.

Four: Non front tabs: rAF is currently not delivered if the page is not the front tab which is great but rAF is an event so even when rAF stops firing because the page is not on the front tab other events still arrive (fetch, onmessage, XHR, websockets, etc...) This means even though your page doesn't get a rAF callback it can still process incoming data (like your chat app's messages).

How is that supposed to work with `gl.commit` loops? It's not the front tab so you want to block the commit so the worker doesn't spin and waste time. If the worker locks then that seems to have implications for all other associated workers and the main thread. If you're using Atomics to sync up things suddenly they'll fail indefinitely even more complicating all the code you have to write to use this feature.



Chrome has already committed to shipping the API. The code as been committed so if nothing changes it will ship automatically in a few weeks with all the issues mentioned above not behind a flag but live so it seems important to understand how to use this and if all these issues were considered and what their solutions are.