[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Public WebGL] WEBGL_dynamic_texture extension proposal

> On 26/07/2012 10:51, David Sheets wrote:
>> The stream producer interface would be useful to any component
>> (standard or custom) that wishes to supply a dynamic,
>> time-synchronized texture stream to a stream consumer ...

> In essence this means this WebGL extension will include extensions for
> other HTML elements. If nobody here objects to it doing so, then in the
> next draft I'll add a stream producer interface which will be used to
> acquire image frames, etc.

I look forward to implementing dynamic texture pipelines sourced from

>>>> The browser has a handful of metrics useful to the page for estimating
>>>> time-to-render latency. ...
>>> Do the browsers currently expose these metrics to JS apps? If so, where is
>>> the API documented?
>> To my knowledge, these metrics are presently unavailable to scripts
>> except through per-page-load benchmarking.
> Is there any documentation anywhere about these metrics? I'm not
> actually sure they are useful in this case but it would be helpful to
> know what they are.

Any formal or informal model of CPU-GPU systems contains these
heuristics. This system modeling is precisely the mandate of The
Khronos Group. Here are a few potential dimensions:

- uncontended API -> GPU command latency (driver, bus, thread dispatch stages)
- uncontended API -> GPU data throughput (driver, bus, thread dispatch stages)
- page composite latency
- video decode latency
- estimated render latency of shader S

No model of these systems will be complete (it's an abstract model...)
but some control more than "I just use 50ms for my consumer latency"
would be nice. Even a single dimension:

- estimated time-to-render for shader S

would be appreciated. I believe the proposal to just guess a latency
was already floated. I would like to guess a latency that has some
bearing to the host's underlying characteristics based on my model of
my specific rendering application.

> Separately but related I was already aware of the issue Florian pointed
> out that the WebGL application does not know when the final page image
> will appear on the screen. I intend to look at previous GL extensions
> that have dealt with similar issues and propose a solution.

There are no guarantees but instead "best effort" for time slices of
shared, asynchronous resources. It seems to be working pretty well for
the net and RTT distribution parameters are successfully employed in
latency compensation for real-time internetwork applications.

>> I am stacking dynamic textures 4 deep sourced from a single video
>> element. V => A => B => C where B and C may depend on any subset of
>> previous producers in the chain.
>> In this use case, HTMLVideoElement V has <=3 different consumer
>> latencies, WebGLRenderingContext A has <=2 consumer latencies, etc.
>> Should 3 separate HTMLVideoElements be created with the same video
>> source? How do I keep them in sync?
> As best I can understand your scenario you are composing several video
> frames to produce a final image. Keeping in mind that the purpose of
> setting the consumer latency is so the producer can keep the audio
> synchronized, the answer to your question is to set the latency to the
> time from acquiring the first image frame to when the final composited
> image is displayed.

How do I get frames at specific temporal spacings?

Declaring latency is a solution for stream synchronization. We have 2+
streams: audio and one or more video streams (possibly from 1 decoder)
but only a way to declare a single stream-stream latency (audio
relative to _this_ dynamic texture acquisition).

How do I synchronize video streams with each other?

>> Sampling does not need to be available for non-RGBA colorspaces
>> (samplerExternalYUV would be abstract and only consumed by
>> conversion). Colorspace conversion branching can be done at
>> compile-time using overloaded convertColorspaceRGBA(...) and sampler
>> type -> sampler type macro renames. With this design, the colorspace
>> sampler type names do not have to be standard -- simply available to
>> the page and overloaded in the GLSL conversion functions. This gives
>> the page author the most control and performance and is a superset of
>> the presently proposed functionality. The author is now free to choose
>> at what time the sampler conversion should be specialized and may
>> anticipate format changes in advance of their occurrence and compile
>> the appropriate shader before it is needed.
> I don't entirely understand what you are proposing but the following
> points must be kept in mind:
>  * The spec. already says that the author's shader receives
>    colorspace-normalized RGBA data when sampling an external texture.

The phases in which the colorspace normalization and per-colorspace
branching occur are unspecified.

>  * The sampler conversion cannot be specialized at compile-time
>      o The video format can change at any time, even within a video
>        when adaptive streaming is used and there is no way for the
>        application to be notified when it changes.

This is known to the decoder. Decoder can say
"samplerExternalOESReallyReallyDynamic". Author can always elect to
use the most generic sampler supertype. No guarantee is made for reuse
of a specialized shader with dynamic texture sources of type outside
supplied sum type.

>      o The value of the sampler variable can be changed at any time via
>        the API

Only samplers of the same type (or compatible subtype) may be bound.
As Acorn elaborated, "You would not create a cubemap texture and then
try to use it as a 3D texture, right?"

>  * We cannot allow general texture operations (copyTexImage,
>    tex{,Sub}Image, etc) to external textures because the hardware
>    generally does not support the necessary conversions when those
>    textures are YUV, even hardware that supports EGLStreams.

Do not overload those API calls for non-RGBA textures.

>  * There is an enormous number of YUV formats. Check out fourcc.org
>    <http://www.fourcc.org/yuv.php>. Generally the format information is
>    only available at very low levels in the software stack (e.g., the
>    OpenMAX IL level). The API's provided for applications to play video
>    (e.g. OpenMAX AL) do not provide the information.  Forcing every
>    author who wants to texture some geometry with video to have to deal
>    with all this is doing them a great disservice.

Function overloading means there is neither forcing to handle
colorspace conversion nor to incur branching in an inconvenient phase.


samplerExternalOES convertColorspaceRGBA(samplerExternalSuperDynamic)
// present proposal implicit in texture2D overload, very conditional
samplerExternalOES convertColorspaceRGBA(samplerExternalQ) // possible
conditional host-specific, fresh sampler type for video format F
samplerExternalOES convertColorspaceRGBA(samplerExternalX) // possible
host-specific, fresh sampler type for shared texture from another

Authors who do not care may always use samplerExternalSuperDynamic
which is effectively the conditional sum of every possible dynamic
texture format decided at the last possible moment.

Authors who care may query their source for a specialized sampler type
and select the conversion branch at the time of their earliest

No colorspace information is leaked. The sampler type names may be
generated fresh for each query. Extremely common dynamic texture
colorspaces may be standardized and overloaded at will (RGBA 8888
seems like a good candidate).

Consider a video pipeline with 3 shader stages. As author, I use
several generic processing stages from a common library. In the
construction of my application, I know I will always use these stages
in a specific configuration. If authors are forced to use an API that
does not give them the ability to specialize their shaders to their
pipeline configuration, they will pay for conditionals over "an
enormous number of YUV formats" at every stage.

Is that a good idea for mobile phones and tablets?

>> TeX is a fine source language. DocBook is a fine target language. No
>> manual conversion is necessary.
>> Many HTML-of-TeX convertors exist not the least of which is HeVeA
>> <http://hevea.inria.fr/>.
>> I'm sure someone in our community would contribute a build system for
>> converting the present TeX source into (X)HTML provided the spec
>> source was made public.
>> Will Khronos publish the TeX source for the specification of the Open Standard?
> The question has never been raised before and therefore never
> discussed.  I personally don't see any reason why not. I'll raise the
> question in Khronos.

Thank you.

> As far as I'm concerned the only reason for using PDF is because you
> have everything in a single file. I do wish the browser makers would
> support MHTML or something like it.

HTML supports inline styles, inline scripts, inline raster images in
data: URLs, and inline vector images with SVG.

What other media is included in Khronos specification documents?

>> Are OpenGL ES extensions drafted in text/plain as they are published?
>> I may or may not already have a parser for this format.
> Yes they are drafted in text/plain. You find the template at
> http://www.opengl.org/registry/doc/template.txt. There was talk of
> moving to DocBook but it hasn't happened.

I'm sure something will turn up.



> Regards
>     -Mark

You are currently subscribed to public_webgl@khronos.org.
To unsubscribe, send an email to majordomo@khronos.org with
the following command in the body of your email:
unsubscribe public_webgl