[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Public WebGL] WEBGL_dynamic_texture redux



Any update on a new draft of this extension?

Intel recently contributed code to WebKit which accelerates texImage2D
taking HTMLVideoElement with a GPU-to-GPU copy when hardware video
decoding is used. See https://bugs.webkit.org/show_bug.cgi?id=111126 .
This should be available for testing in the current Chrome Canary and
should work at least on Windows and Mac OS. It would be great to do
some performance comparisons between Chrome Stable and Canary for
video texturing applications.

-Ken



On Tue, Mar 26, 2013 at 8:24 PM, Mark Callow <callow.mark@artspark.co.jp> wrote:
> Hi Florian,
>
> Thanks for your suggestions on how to resolve those issues.
>
> There are other issues. In fact I have a new draft in progress with several
> major changes; most of those are reflected in the example code I posted a
> while ago. I have been working on something else but I expect to finish that
> within the next day. I will then be able to spend some time back on dynamic
> textures. I'll finish the draft as quickly as I can.
>
> Regards
>
>     -Mark
>
>
>
> On 2013/03/26 4:38, Florian Bösch wrote:
>
> http://www.khronos.org/registry/webgl/extensions/proposals/WEBGL_dynamic_texture/
> lists issues 7 9, 10, 12 and 13 as unresolved.
>
> - Are there any other issues than those 13?
> - Have any of the issues been resolved?
>
> I would suggest the following resolutions:
>
> 7) Accept the suggestion presented.
> 10) No. This extension should provide a solution for video.
>
> To issues 12 and 13: Is there any solution to these issued raised with the
> time flavor to use?
>
> To issue 11: Does it make this specification appear faster if it is defined
> as "core" rather than as extension, in the 2 years that this issue has not
> been resolved? If so, then I will suggest yes. If no then I will suggest no.
> Anybody?
>
>
>
> On Wed, Nov 28, 2012 at 4:32 AM, Mark Callow <callow.mark@artspark.co.jp>
> wrote:
>>
>> On 2012/11/22 11:15, David Sheets wrote:
>>
>>
>>
>>   //   // readonly int freq; ? do videos get one per channel? only max
>> frequency of all media streams?
>>
>> What would you use this for? Without knowing the use case, the obvious
>> answer is the frequency (framerate) of the producer video stream. However
>> several modern video formats do not have a fixed framerate.
>>
>> I would use this for relating frame counts to
>> latencies/timeouts/clocks. Specifically, without knowing the inherent
>> maximum frequency of the producer, the consumer must guess its frame
>> budget initially and back-off causing A/V glitches.
>>
>> I agree this would be useful for exactly the reasons you cited.
>> Unfortunately web browsers do not make the frame-rate available to scripts
>> even though most, if not all, container formats store this information. For
>> example WebM supports the same FrameRate item as the Matroska container.
>>
>> Ideally we would persuade browser vendors/WHATWG to add it as an attribute
>> of the HTMLVideoElement. Alternatively, since there may not be any uses
>> outside WebGL, we could add it to the WebGLDynamicTextureSource interface,
>> if we go that route.
>>
>>
>> Are you planning on exposing the variable framerate of decoders to
>> stream consumers in the msc? Is it bad to insert "duplicate" frames
>> and increment the msc when the source framerate drops below max? I
>> think consistency is valuable here and simplifies the author's work.
>> If I test my generic video app with constant framerate videos but not
>> variable framerate videos, I might be disappointed with performance on
>> sophisticated codecs in future. Can this consistency abstraction leak
>> in some way I am missing?
>>
>> I have come to the opinion that msc is not a useful concept for a
>> variable-frame-rate video source. I'm leaning towards exposing the time at
>> which the frame is expected to be displayed.
>>
>> I'll come back to your other questions in a separate message. I want to
>> keep this focused on  interface for hooking up producers.
>>
>> I am thinking of the following way of setting up a dynamic texture modeled
>> on your WebGLDynamicTextureSource proposal.
>>
>>   // Use object for "global" variables to avoid polluting the global
>> space.
>>   var g = {};
>>
>>   // Array of files currently loading
>>   g.loadingFiles = [];
>>
>>     gl.dte = gl.getExtension("WEBGL_dynamic_texture");
>>     g.videoTexture = {
>>       texture: gl.createTexture(),
>>       video: createVideoElement(),
>>       stream: null
>>     };
>>     // gl.activeTexture(gl.TEXTURE0);
>>     gl.bindTexture(gl.TEXTURE_EXTERNAL_OES, g.videoTexture.texture);
>>     // Create stream with currently bound texture as consumer.
>>     g.videoTexture.stream = gl.dte.createStream();
>>     connectStreamToProducerAndSource(stream, video,
>> "http://myserver/myvideo.mp4";);
>>
>> where createVideoElement() is
>>
>>   // Create a hidden video element and append to the document
>>   // XXX Is it necessary to append the element?
>>   function createVideoElement() {
>>     var video = document.createElement('video');
>>     video.autoplay = true;
>>     video.controls = false;
>>     video.hidden = true;
>>     document.body.appendChild(video);
>>     return video;
>>   }
>>
>> Note, creation of the video element was missing from the example
>> previously.
>>
>> connectStreamToProducerAndSource() is
>>
>>   function connectStreamToProducerAndSource(stream, video, url) {
>>
>>
>>     video.onload = function() {
>>       g.loadingFiles.splice(g.loadingFiles.indexOf(video), 1);
>>     };
>>     video.onerror = function() {
>>       g.loadingFiles.splice(g.loadingFiles.indexOf(video), 1);
>>       // Notify user? Abort application?
>>       // video.error gives the reason.
>>     };
>>     g.loadingFiles.push(video);
>>     video.wssConnectToSourceAsProducer(url, stream);
>>   }
>>
>> You first create the stream with its consumer then pass that to
>> wssConnectToSourceAsProducer method that is part of the interface added to
>> HTMLVideoElement. I'd like to hear from the WebIDL and browser experts
>> whether something like
>>
>> interface WebGLStreamSource {
>>     void wssConnectToSourceAsProducer(DomString url, WebGLStream stream);
>>     void wssDisconnectFromStream();
>>     unsigned long wssFrameRate;
>> };
>> HTMLVideoElement implements WebGLStreamSource;
>>
>> is acceptable.
>>
>> Suggestions for better names are welcome.
>>
>> Behind this are the concepts that
>>
>> A producer can only be connected to a single stream
>> The stream must be connected to the consumer before the stream is passed
>> to the producer so the browser or underlying EGLStream implementation knows
>> about both ends of the stream before it creates the video decoder.
>>
>> For this same reason the producer element cannot have done any processing
>> of the video file before the connection is made, so the src attribute must
>> be empty.
>>
>> DisconnectFromStream is used for handling context lost events. It will
>> abort any in-progress connection attempt as well as disconnecting an
>> established connection.
>> The "wss" prefix is to identify that these properties are part of the
>> WebGLStreamSource interface not the basic element properties.
>>
>> Having a standard interface for producers is attractive but I'm unsure how
>> best to append this to existing elements as I'm lacking both JavaScript and
>> WebIDL expertise.
>>
>> Opinions?
>>
>> Regards
>>
>>     -Mark
>>
>> --
>> 注意:この電子メールには、株式会社エイチアイの機密情報が含まれている場合が有ります。正式なメール受信者では無い場合はメール複製、
>> 再配信または情報の使用を固く禁じております。エラー、手違いでこのメールを受け取られましたら削除を行い 配信者にご連絡をお願いいたし ます.
>>
>> NOTE: This electronic mail message may contain confidential and privileged
>> information from HI Corporation. If you are not the intended recipient, any
>> disclosure, photocopying, distribution or use of the contents of the
>> received information is prohibited. If you have received this e-mail in
>> error, please notify the sender immediately and permanently delete this
>> message and all related copies.
>
>
>
>
>
>
> Regards
>
>     -Mark
>
> --
> 注意:この電子メールには、株式会社エイチアイの機密情報が含まれている場合が有ります。正式なメール受信者では無い場合はメール複製、
> 再配信または情報の使用を固く禁じております。エラー、手違いでこのメールを受け取られましたら削除を行い配信者にご連絡をお願いいたし ます.
>
> NOTE: This electronic mail message may contain confidential and privileged
> information from HI Corporation. If you are not the intended recipient, any
> disclosure, photocopying, distribution or use of the contents of the
> received information is prohibited. If you have received this e-mail in
> error, please notify the sender immediately and permanently delete this
> message and all related copies.

-----------------------------------------------------------
You are currently subscribed to public_webgl@khronos.org.
To unsubscribe, send an email to majordomo@khronos.org with
the following command in the body of your email:
unsubscribe public_webgl
-----------------------------------------------------------