[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Public WebGL] WEBGL_texture_from_depth_video extension proposal



On Thu, Nov 6, 2014 at 9:08 AM, Kenneth Russell <kbr@google.com> wrote:
The reason to expose an extension from my point of view is (a) to allow detection of support for uploading depth videos to WebGL and (b) to provide a single, efficient upload path that users can rely on.
a) Is support depending on anything else than the browser implementing it?

b) Is that goal best served with an extension?

Since depth texture streaming interacts with various other specifications (canvas, typed arrays and webgl 1, 2, 2.1, 3, depth textures, floating point textures and <video>), it would seem to me the most consistent and easiest to use entry point for any discovering support and consistently supporting a feature would be the media capture depth specification, rather than modify half a dozen other specifications and add pieces to it.
 
3) The style of how to support the upload isn't very friendly to users because it requires manual conversion in the fragment shader. I'd much rather see float/half-float texture formats being satisfied, where the browser knows about the surface format of the depth texture, and converts it (on GPU, without blocking operation) to the specified floating point format for upload. This'd have the advantage of offering users a transparent support for depth images, regardless of their underlying format.

Yes, with WebGL 1.0 the conversion is a little painful. This will become trivial with WebGL 2.0 and the availability of R16UI / R16I textures. I'd like to focus on getting WebGL 2.0 out the door, but we'd appreciate any concrete suggestions you have for the Media Capture folks about better data upload formats to use with WebGL 1.0.

-Ken


There's some concern for me for backwards/forwards compatibility and extension proliferation. The current extension specification defines as only supported format upload format UNSIGNED_SHORT_5_6_5.

What if...
  • A new device appears that delivers depth fields as UNSIGNED_BYTE --> WEBGL_texture_from_depth_video_unsigned_byte?
  • A new device appears that delivers depth fields as UNSIGNED_INT -> WEBGL_texture_from_depth_video_unsigned_integer?
  • A new device appears that delivers depth fields as Float -> WEBGL_texture_from_depth_video_float32?
  • A new texture format should be supported (float) -> WEBGL_texture_from_depth_video_to_float?
  • A new texture format should be supported (depth_component) -> WEBGL_texture_from_depth_video_depth_component?
  • WebGL 2.0's texture formats should be supported -> WEBGL_texture_from_depth_video2?
  • WebGL 2.1's texture formats should be supported -> WEBGL_texture_from_depth_video2_1?
  • etc.
I'd like depth upload to work for the following WebGL 1 formats:
  • gl.RGB, gl.UNSIGNED_BYTE -> greyscale 0 - 255
  • gl.RGBA, gl.UNSIGNED_BYTE -> greyscale 0 - 255, alpha 1
  • gl.RGBA, gl.FLOAT -> greyscale 0 - 1 (32-bit float), alpha 1
  • gl.RGB, gl.FLOAT -> greyscale 0 - 1 (32-bit float)
  • gl.LUMINANCE, gl.UNSIGNED_BYTE -> 0 - 255
  • gl.LUMINANCE, gl.FLOAT -> 0-1
  • gl.RGBA, ext.HALF_FLOAT_OES -> 0-1 (16-bit float), alpha 1
  • gl.RGB, ext.HALF_FLOAT_OES -> 0-1 (16-bit float)
  • ext.DEPTH_COMPONENT, gl.UNSIGNED_SHORT -> 0-2^16-1
  • ext.DEPTH_COMPONENT, gl.UNSIGNED_INT -> 0-2^32-1
  • ext.DEPTH_COMPONENT, ext.UNSIGNED_INT_24_8_WEBGL -> 0-2^24-1, stencil 0
I'd also like to see depth uploads work for the about 30 internal formats that is has, including things that where in extensions in WebGL 1 and are now core in WebGL 2.

I'd also like to see depth uploads work with any of the changed texture formats coming to WebGL 2.1 and WebGL 3.0.

And I'd like this to work without having to introduce 3 dozen new extensions to make it happen in the years to come...