[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Public WebGL] Gamma correction and texImage2D/texSubImage2D



Thatcher Ulrich wrote:
> On Sat, Sep 4, 2010 at 9:45 PM, Steve Baker <steve@sjbaker.org> wrote:
>   
>> I agree that there are undoubtedly going to be issues with roundoff
>> error and precision.  In the short term, I'll do what I've always done -
>> instructed my artists to paint textures in 'linear' color space and to
>> have their monitors gamma color-calibrated every six months to ensure
>> that they're seeing those textures optimally.
>>     
>
> What format & depth are your artists saving their work in?  And then
> what texture format(s) do you convert to?  How are you planning to get
> this data into WebGL?
>   
Well - I have to answer this two ways - because I'm doing two jobs.

* In my (paying) day job - I am the graphics lead for Total Immersion
(www.totimm.com) - but I saw the same practices when I worked for Midway
Games and in many games & simulation companies before that.  I'm writing
D3D (yuk!) graphics engines for 'serious games' for things like training
firefighters and the pilots of unmanned drone aircraft and such.  The
artists use 8/8/8 or 8/8/8/8 PNG or TIFF (some GIS terrain tools use it)
with a gamma of 1.0 (ie, linear color space) and we convert to DDS for
loading into the game.  DDS supports DXT1/3/5 compression and
no-compression which gives us the choice to use lossy compression where
space is critical.  We also have the advantage of stating 'minimum
hardware specs' which means we never have to deal with cellphones or
hardware that doesn't support full floating point textures and shaders. 
One day we'd LOVE to be able to use WebGL to support this stuff - but
we're far from there yet.   Part of the reason I'm following this
mailing list is that I want to be sure that WebGL could ultimately
support serious simulation engines and run AAA quality video games.

* In my spare time (hahahah!) I'm starting with a small, dedicated (and
as yet unpaid) team to put together an experimental set of  WebGL-based
games to see whether we can make money using adverts and T-shirt sales
and in-game revenue (pay us a dollar and get that neat weapon you always
wanted!).   We currently use 8/8/8 and 8/8/8/8 PNG with a gamma of 1.0
(ie linear color space) and do no further compression.  Having looked
carefully at ETC1, and kicked around some ideas with the guy at Ericsson
who invented it, it's clear that it may prove useful for some kinds of
data - but I'm still a little skeptical about it in general.  There may
be some super-devious shaderly tricks to make it do things it was never
designed to do...but that's still a matter of investigation for me.

In the latter case, my biggest concern is with platforms that may not
support 8/8/8 or 8/8/8/8 formats internally and which may crunch them
down to 5/6/5 or 5/5/5/1 or 4/4/4/4 formats internally.  Since I use
texture in 'non-traditional' ways - I'm having to get creative about
only using the high order 4 bits in some situations.  It's painful.  So
my best guess right now is that I'll be using the built-in file loaders
with linear color space PNG.
> There are some relevant constraints on WebGL:
>
> * browsers (currently) work with 8-bit sRGB_Alpha color buffers, so
> that's the format that WebGL output ends up in.  I don't think WebGL
> 1.0 can realistically spec anything else for output.  (In the future,
> perhaps browsers will be able to handle linear color spaces at higher
> bit depths, but I don't know if anybody is seriously considering that
> yet, so I doubt there's any point in spec'ing anything in WebGL 1.0.)
>   
That's great.
> * WebGL has texImage2D calls that take raw buffers of data.  Currently
> (default) behavior passes this data straight to the texture formats,
> so any automatic gamma treatment would be a change.
>   
Yes - that is true.  I think we need the specification to say that you
CAN automatically convert color spaces when you do this - but not that
you MUST.  That's necessary, not least because with low end cellphone
hardware, repeatedly converting back and forth between color spaces will
introduce so much rounding error that it'll be unusable - so
applications will want to disable that (even if it's "mathematically
required") in order to get the lesser of two evils.

However, IMHO, the specification needs to support (at least in theory)
mathematical correctness and must never impos**e  incorrectness because
by the time this specification becomes obsolete, we'll probably have 12
or 16 bit integer and 16 or 32 bit floating point per component (I
already use 32/32/32/32 for HDR lighting in some places in my "day job"
graphics engine) and doing this conversion will ultimately be entirely
painless.
> * browsers can load image formats in PNG and JPEG, which are most
> typically 8-bit sRGB.  WebGL behavior when using these formats is
> definitely worth spec'ing.
>   
Yes.
> * PNG has the ability to specify a per image gamma value (the gAMA
> thing referenced earlier).  Browsers appear to handle this
> differently.  see this reference page, at "Images with gamma chunks":
> http://www.libpng.org/pub/png/pngsuite.html  On the Mac I'm using
> right now, Chrome and Safari do not do gamma correction, while Firefox
> does.  You can also clearly see the quantization errors in the Firefox
> images with the lower gamma values.  The Chrome and Safari behavior is
> (arguably) a bug.
>   
Yes, I agree.
> * PNG has the ability to store 16-bit color depth.  However, my
> understanding is that current browsers take all input images
> (including PNG images with 16-bit color depth) and convert them to
> sRGB_Alpha internally, before WebGL has a chance to see the data.
> Also, the WebGL spec does not appear to have any texture formats that
> have more than 8 bits per color component.  This would be a great
> thing to improve, post WebGL 1.0, since hi-fi WebGL apps could make
> good use of it.
>   
Absolutely.  Supporting (in particular) floating point textures would be
a big win...but there are many desktop/laptop chipsets that can't do
that - and I fear it'll be a good few years before cellphones can do
that.   But this is doable as an extension.  The issues of color space
correctness are the underpinnings of the specification and should be
handled rigorously from the get-go because inserting them later would be
tortuous and disruptive.
> It seems to me there are two unresolved questions for WebGL 1.0
>
> 1) Should WebGL attempt to nail down how browsers are supposed to
> handle PNG's with a non-default gAMA value?  Viable options here are:
>
>   a) leave it up to the browser (status quo, but behavior may differ
> among browser; in practice apps will have to supply sRGB data and do
> any additional conversions themselves).
>   
Not tolerable.  Because textures are very often used for storing things
other than images and writing our own texture loaders in JavaScript is
ridiculous - doing any kind of automatic color space conversion without
a way to turn it off would make WebGL useless for all but the simplest
kinds of 3D graphics.   Worse still, because JavaScript is slow, we have
to push more work into the GPU than on a traditional OpenGL or D3D
platform - that INCREASES the number of situations where we use texture
"non-traditionally" in order to get good performance.  If this were the
choice - I'd stop work on my WebGL games.
>   b) demand conversion to sRGB_Alpha based on PNG metadata (i.e.
> converge on current Firefox behavior, however non-sRGB behavior will
> be a corner case for browsers and smart WebGL developers may opt to
> always supply sRGB data and do any conversions themselves)
>
>   c) demand passing raw data straight through.  16-bit components
> would be rounded or truncated to 8-bit.  (i.e. converge on current
> WebKit behavior, similar caveats as option b)
>   
d) Have the WebGL texture file loader convert whatever color space the
file is in - to a uniform linear color space.  With the option to turn
that conversion off for files that contain non-traditional (non-image)
data - and in cases where the roundoff error inherent in the conversion
is not acceptable - or when the application knows that the source data
is in linear color space regardless of what the file header happens to say.

i.e. Support both and let the application decide.
> 2) Should WebGL add a PixelStore option that does some kind of gamma
> conversion?  (Where the thread started.)  IMO the status quo (do
> nothing) is pretty much fine.  Apps that want a specific
> interpretation of their image data can either pre-process it so the
> raw data matches what they want in texture RAM, or else do custom
> processing via GPU with the existing facilities.
>   
Since we've now established (I hope!) that the canvas spec requires that
the linear color space WebGL canvas must be converted into 'device color
space' at some point before it hit the screen (which for us probably
means "it's gamma corrected in the compositor") - there is absolutely no
reason to ever want to do this.  Automatically converting linear-space
textures into gamma space then converting the rendering results into
gamma space would guarantee an ugly mess on the output.  That's a waste
of CPU time, implementation effort and it's mathematically indefensible.
>>> I believe EXT_texture_sRGB is designed to make it easier to be "linear
>>> correct" under these circumstances.  It basically allows you to tell
>>> OpenGL that your 8-bit texture data is in sRGB format, and that you
>>> want OpenGL to an 8-bit-sRGB to higher-precision-linear-RGB conversion
>>> when sampling the data from a shader.  Your shader will operate on
>>> linear data (in floating point format), and the frame buffer is still
>>> your shader's problem
>>> Yes - but it's an extension that we can't rely on always being
>>> implemented (I doubt ANGLE could emulate it either).
>>>       
>
> It can be trivially implemented on any hardware that has an internal
> texture format with at least 12 bits per component; just convert the
> data to linear and store in the higher-depth format.  The practical
> problem is that it wastes texture RAM, hence the preference to have
> lookup tables in the GPU.
>   
But we're potentially operating with hardware that may not even support
8 bits per component let alone 12!  So it certainly cannot be "trivially
implemented" on all WebGL clients.  Hence we certainly can't rely on it
- and most certainly we can't write our specification based upon it!

Worse still, implementing it simply with "lookup tables in the GPU"
(which, I'll grant that the extension spec allows) means that the sRGB
texels are linearly interpolated for GL_LINEAR and
GL_LINEAR_MIPMAP_LINEAR textures - and that's wrong!   When you minify a
texture, the hardware is (in effect) calculating the contributions of
four texels from each of two MIP levels.   In linear color space, that's
a simple lerp operation that's super-cheap to do in hardware - but in
sRGB, doing that that gives too much weight to the bright texels and not
enough to the dark ones.  The consequences are that the texture will
alias along bright-to-dark transitions.

I can't imagine many GPU manufacturers building proper sRGB
interpolators into the highest bandwidth part of their engines - so I'll
be surprised if many of them support this extension "the right
way"...which makes it all but useless to people who want nice-looking
graphics.

  -- Steve

-----------------------------------------------------------
You are currently subscribed to public_webgl@khronos.org.
To unsubscribe, send an email to majordomo@khronos.org with
the following command in the body of your email: