[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Public WebGL] texImage2D on unfinished images and video



On Mon, Oct 22, 2012 at 11:28 AM, Gregg Tavares (社用) <gman@google.com> wrote:
> So checked. Three.js always uses texImage2D for video
> So does this
> http://dev.opera.com/static/articles/2012/webgl-postprocessing/webgl-pp/multipass.html
> and this
> http://badassjs.com/post/16472398856/photobooth-style-live-video-effects-in-javascript-and
> and this
> https://developer.mozilla.org/en-US/docs/WebGL/Animating_textures_in_WebGL
> and this http://videos.mozilla.org/serv/mozhacks/flight-of-the-navigator/
> and this http://radiapp.com/samples/Video_HPtrailer_WebGL_demo.html
> and this http://sp0t.org/videoriot/
>
> This one use texSubImage2D
> http://webglsamples.googlecode.com/hg/color-adjust/color-adjust.html
>
> I only bring that up to point out that the fact that texSubImage2D usage
> would be problematic may not be that important
>
> On the other hand, both video and WebRTC are planning on dynamic resolution
> adjustments based on how fast your internet connection is.
>
> I bring that up because it means knowing the resolution so you can call
> texSubImage2D seems like something you can only count on if you have 100%
> control over the video source.
>
> I seems like to help WebGL fit more uses cases (cases where you don't
> control those sources like say a video site or a video chat site) that it
> would be more useful if things "just worked" which is relatively easy if
> texImage2D just works and seems like it would be rather painful if it could
> possibly generate an error in various states of switching resolutions or
> changing videos.
>
> I feel like GetTexLevelParameter is an orthogonal issue. There are plenty of
> things you can do without knowing the resolution of the image/video.
>
> Also, while an unloaded video/image could return 1x1, off the top of my
> head, what it returns while changing resolutions or queuing the next video
> could be implementation defined. Some browsers might return 1x1 while
> switching. Others might return the last valid frame until such time as a
> frame from the new source is ready. As long as there is no error that seems
> better to me.

It seems that the consensus is to change the WebGL specification as you suggest.

The changes would essentially be:

  - If an incomplete HTMLImageElement or HTMLVideoElement is passed to
either texImage2D or texSubImage2D, then the source is treated as a
1x1 texture containing the RGBA value (0, 0, 0, 1). This means that
texImage2D will cause the level of the passed texture object to be
redefined as size 1x1; texSubImage2D will upload a single black pixel
at the given xoffset and yoffset. It would be defined that an OpenGL
error would never occur as a result of these calls.

Are there any objections to these changes? If not, I'll update the spec.

-Ken



>
> On Mon, Oct 22, 2012 at 10:32 AM, Kenneth Russell <kbr@google.com> wrote:
>>
>> On Sun, Oct 21, 2012 at 11:52 PM, Gregg Tavares (社用) <gman@google.com>
>> wrote:
>> > My concern is if you have a video player library that plays a playlist
>> > of
>> > videos and then you decide later to add some WebGL to it, you have to go
>> > through all kinds of contortions just to avoid an INVALID_OPERATION. The
>> > framework you're using for the video playlists seems like it would be
>> > unlikely to have the needed functionality built in to let you know when
>> > top
>> > stop calling texImage2D and when to start again.
>> >
>> > If the error was removed and 1x1 was substituted that problem would go
>> > away
>> >
>> > I don't see what error adds except to make it more complicated.
>>
>> Let's think through this use case. The playlist library iterates
>> through different videos. As it switches between videos it's clearly
>> necessary to send some sort of event to the application, because the
>> videos might be different resolutions. Unless the application is
>> calling texImage2D all the time (instead of texSubImage2D most of the
>> time), it will have to watch for these events to figure out when it
>> might need to reallocate the texture at a different size. (If the app
>> calls texImage2D all the time, then it doesn't matter whether WebGL
>> generates INVALID_OPERATION or a 1x1 black texture. The app will never
>> need to know whether the upload succeeded or failed, because failures
>> will be transient, and the app will automatically recover from them
>> when the next video starts playing. If texImage2D generates
>> INVALID_OPERATION, the previous contents of the texture will be
>> preserved.)
>>
>> If WebGL generates INVALID_OPERATION for incomplete texture uploads,
>> then the app either needs to watch for those errors coming back from
>> OpenGL, or watch for the onplay event to know when it's guaranteed to
>> be OK to upload a frame. If the app doesn't do this, and mostly calls
>> texSubImage2D to upload new frames, then the video can get "stuck" as
>> the app tries to upload new frames via texSubImage2D while the
>> underlying texture is the wrong size.
>>
>> If WebGL silently creates a 1x1 black texture for incomplete texture
>> uploads, then the app still needs to watch for the onplay event to
>> know when to call texImage2D instead of texSubImage2D. Otherwise, the
>> video can still get "stuck" displaying a 1x1 black texture.
>>
>> As far as I see it, the app has to do the same work regardless of
>> whether incomplete texture uploads generate an error or silently
>> produce a 1x1 black texture. The main difference is that the error can
>> be observed by the application, but the silent allocation of the 1x1
>> black texture can not, because glGetTexLevelParameter doesn't exist in
>> the GLES API. For this reason I continue to think that generating
>> INVALID_OPERATION is the more transparent behavior for WebGL.
>>
>> Regardless of the decision here, I agree with you that the behavior
>> should be specified and we collectively should try to write some tests
>> for it -- ideally, ones that explicitly force failure or pauses at
>> certain download points, rather than just trying to load and play the
>> video many times.
>>
>> -Ken
>>
>>
>>
>> > On Fri, Oct 19, 2012 at 3:09 PM, Kenneth Russell <kbr@google.com> wrote:
>> >>
>> >> On Tue, Oct 16, 2012 at 4:46 PM, Gregg Tavares (社用) <gman@google.com>
>> >> wrote:
>> >> >
>> >> >
>> >> >
>> >> > On Tue, Oct 16, 2012 at 4:28 PM, Kenneth Russell <kbr@google.com>
>> >> > wrote:
>> >> >>
>> >> >> On Tue, Oct 16, 2012 at 3:54 PM, Gregg Tavares (社用)
>> >> >> <gman@google.com>
>> >> >> wrote:
>> >> >> > I don't think the spec makes this clear what happens when you try
>> >> >> > to
>> >> >> > call
>> >> >> > texImage2D or texSubImage2D on an image or video that is not yet
>> >> >> > loaded.
>> >> >>
>> >> >> Right, the behavior in this case is not defined. I'm pretty sure
>> >> >> this
>> >> >> was discussed a long time ago in the working group, but it seemed
>> >> >> difficult to write a test for any defined behavior, so by consensus
>> >> >> implementations generate INVALID_OPERATION when attempting to upload
>> >> >> incomplete images or video to WebGL textures.
>> >> >
>> >> >
>> >> > I don't see a problem writing a test. Basically
>> >> >
>> >> > var playing = false
>> >> > video.src = url
>> >> > video.play();
>> >> > video.addEventListener('playing', function() { playing = true;});
>> >> >
>> >> > waitForVideo() {
>> >> >   gl.bindTexture(...)
>> >> >   gl.texImage2D(..., video);
>> >> >   if (playing) {
>> >> >      doTestsOnVideoContent();
>> >> >      if (gl.getError() != gl.NO_ERROR) { testFailed("there should be
>> >> > no
>> >> > errors"); }
>> >> >   } else {
>> >> >     requestAnimationFrame(waitForVideo);
>> >> > }
>> >> >
>> >> > This basically says an implementation is never allowed to generate an
>> >> > error.
>> >> > it might not be a perfect test but neither is the current one. In
>> >> > fact
>> >> > this
>> >> > should test it just fine
>> >> >
>> >> >   video = document.createElement("video");
>> >> >   gl.texImage2D(..., video);
>> >> >   glErrorShouldBe(gl.NO_ERROR);
>> >> >   video.src = "someValidURL":
>> >> >   gl.texImage2D(..., video);
>> >> >   glErrorShouldBe(gl.NO_ERROR);
>> >> >   video.src = "someOtherValidURL":
>> >> >   gl.texImage2D(..., video);
>> >> >   glErrorShouldBe(gl.NO_ERROR);
>> >> >
>> >> > Then do the other 'playing' event tests.
>> >> >
>> >> >>
>> >> >>
>> >> >> > Example
>> >> >> >
>> >> >> >    video = document.createElement("video");
>> >> >> >    video.src = "http://mysite.com/myvideo";;
>> >> >> >    video.play();
>> >> >> >
>> >> >> >    function render() {
>> >> >> >      gl.bindTexture(...)
>> >> >> >      gl.texImage2D(..., video);
>> >> >> >      gl.drawArrays(...);
>> >> >> >      window.requestAnimationFrame(render);
>> >> >> >   }
>> >> >> >
>> >> >> > Chrome right now will synthesize a GL error if the system hasn't
>> >> >> > actually
>> >> >> > gotten the video to start (as in if it's still buffering).
>> >> >> >
>> >> >> > Off the top of my head, it seems like it would just be friendlier
>> >> >> > to
>> >> >> > make a
>> >> >> > black texture (0,0,0,1) or (0,0,0,0) 1x1 pixels for videos that
>> >> >> > aren't
>> >> >> > loaded yet.
>> >> >> >
>> >> >> > Otherwise, having to check that the video is ready before calling
>> >> >> > texImage2D
>> >> >> > seems kind of burdensome on the developer. If they want to check
>> >> >> > they
>> >> >> > should
>> >> >> > use video.addEventListener('playing') or similar. If they were
>> >> >> > making
>> >> >> > a
>> >> >> > video player they'd have to add a bunch of logic when queuing the
>> >> >> > next
>> >> >> > video.
>> >> >> >
>> >> >> > Same with images.
>> >> >> >
>> >> >> >    img = document.createElement("img");
>> >> >> >    img.src = "http://mysite.com/myimage";;
>> >> >> >
>> >> >> >    function render() {
>> >> >> >      gl.bindTexture(...)
>> >> >> >      gl.texImage2D(..., img);
>> >> >> >      gl.drawArrays(...);
>> >> >> >      window.requestAnimationFrame(render);
>> >> >> >   }
>> >> >> >
>> >> >> > If you want to know if the image has loaded use img.onload but
>> >> >> > otherwise
>> >> >> > don't fail the call?
>> >> >> >
>> >> >> > What do you think? Good idea? Bad idea?
>> >> >>
>> >> >> My initial impression is that this change is not a good idea. It
>> >> >> would
>> >> >> expose the specification, implementations and applications to a lot
>> >> >> of
>> >> >> corner case behaviors. For example, if a video's width and height
>> >> >> hasn't been received yet, then texImage2D(..., video) would have to
>> >> >> allocate a 1x1 texture; but if the width and height are known, then
>> >> >> it
>> >> >> would have to allocate the right amount of storage. A naive app
>> >> >> might
>> >> >> call texImage2D only the first time and texSubImage2D subsequently,
>> >> >> so
>> >> >> if the first texImage2D call was made before the metadata was
>> >> >> downloaded, it would never render correctly. I think the current
>> >> >> fail-fast behavior is best, and already has the result that it
>> >> >> renders
>> >> >> a black texture if the upload fails; the call will (implicitly)
>> >> >> generate INVALID_OPERATION, and the texture will be incomplete.
>> >> >
>> >> >
>> >> > I don't see how that's worse than the current situation which is you
>> >> > call
>> >> > texImage2D and pray it works. You have no idea if it's going to work
>> >> > or
>> >> > not.
>> >> > If you do this does it work?
>> >> >
>> >> >   okToUseVideo = false;
>> >> >   video = document.createElement("video");
>> >> >   video.src = "movie#1"
>> >> >   video.addEventListener('playing', function() { okToUseVideo = true;
>> >> > }
>> >> >
>> >> >   frameCount = 0;
>> >> >
>> >> >   function render() {
>> >> >      if (okToUseVideo) {
>> >> >         gl.texImage2D(...  , video);
>> >> >
>> >> >         ++frameCount;
>> >> >         if (frameCount > 1000) {
>> >> >           video.src = "movie2";
>> >> >         }
>> >> >      }
>> >> >   }
>> >> >
>> >> >
>> >> > Basically after some amount of time I switch video.src to a new
>> >> > movie.
>> >> > Is
>> >> > the video off limits now?
>> >>
>> >> The assumption should be "yes". When the source of the media element
>> >> is set, it is immediately invalid to use until the onload / playing
>> >> handler is called.
>> >>
>> >> > Will it use the old frame from the old movie until
>> >> > the new movie is buffered or will it give me INVALID_OPERATION? I
>> >> > have
>> >> > no
>> >> > idea and it's not specified. Same with img.
>> >> >
>> >> > img.src = "image#1"
>> >> > img.onload = function() {
>> >> >   img.src = "image#2";
>> >> >   gl.texImage2D(...., img);  // what's this? old image, no image,
>> >> > INVALID_OPERATION?
>> >> > }
>> >>
>> >> Yes, this should be assumed to produce INVALID_OPERATION.
>> >>
>> >>
>> >> > The texSubImage2D issue is not helped by the current spec. If
>> >> > video.src
>> >> > =
>> >> > useChoosenURL then you have no idea what the width and height are
>> >> > until
>> >> > the
>> >> > 'playing' event (or whatever event) which is no different than if we
>> >> > changed
>> >> > it.
>> >> >
>> >> > Changing it IMO means far less broken websites and I can't see any
>> >> > disadvantages. Sure you can get a 1x1 pixel texture to start and call
>> >> > texSubImage2D now but you can do that already.
>> >>
>> >> With the current fail-fast behavior, where INVALID_OPERATION will be
>> >> generated, the texture will be in one of two states: (1) its previous
>> >> state, because the texImage2D call failed; or (2) having the width and
>> >> height of the incoming media element.
>> >>
>> >> With the proposed behavior to never generate an error, the texture
>> >> will be either 1x1 or width x height. Another problem with this
>> >> proposal is that there is no way with the ES 2.0 or WebGL API to query
>> >> the size of a level of a texture, because GetTexLevelParameter was
>> >> removed from the OpenGL ES API (and, unfortunately, not reintroduced
>> >> in ES 3.0). Therefore the behavior is completely silent -- there is no
>> >> way for the application developer to find out what happened (no error
>> >> reported, and still renders like an incomplete texture would).
>> >>
>> >> I agree that the failing behavior should be specified and, more
>> >> importantly, tests written verifying it. If the INVALID_OPERATION
>> >> error were spec'ed, would that address your primary concern? I'm not
>> >> convinced that silently making the texture 1x1 is a good path to take.
>> >>
>> >> -Ken
>> >>
>> >>
>> >> >>
>> >> >>
>> >> >> If we want to spec this more tightly then we'll need to do more work
>> >> >> in the conformance suite to forcibly stall HTTP downloads of the
>> >> >> video
>> >> >> resources in the test suite at well known points. I'm vaguely aware
>> >> >> that WebKit's HTTP tests do this with a custom server. Requiring a
>> >> >> custom server in order to run the WebGL conformance suite at all
>> >> >> would
>> >> >> have pretty significant disadvantages.
>> >> >>
>> >> >> -Ken
>> >> >
>> >> >
>> >
>> >
>
>

-----------------------------------------------------------
You are currently subscribed to public_webgl@khronos.org.
To unsubscribe, send an email to majordomo@khronos.org with
the following command in the body of your email:
unsubscribe public_webgl
-----------------------------------------------------------