[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Public WebGL] Support precompiled shaders as extensions



I think the problem is that we rely on the shader compilation step to
avoid certain shader "exploits" that could be problematic for web-based
use.

So some things that are legal for native code shaders are not legal for
WebGL shaders for this reason.  Allowing binary shaders would provide a
means for evil-doers to sneak in things that we regard as illegal.

Giving up our control of that is risky.

We could (in theory) produce our own binary shader language and have that
be converted to the native language on load - but that would bring back
some sort of conversion/checking step that would also become slow over
time.

The idea of having the browser cache "trusted" binary shaders is a good
thing - but it's only going to help on the second and subsequent times you
visit some particular website.

Ideally, we just need faster shader compilers..."just in time" shader
compilers...that kind of thing.

After all, WebGL is driven from JavaScript - and that's processed by the
browser efficiently enough to not be a problem.  We're not demanding that
JavaScript be pre-compiled into x86 machine-code blobs.

  -- Steve



Maksims Mihejevs wrote:
> Thank you Florian on coming back on that.
>
> It is important to solve this problem earlier, as solution won't apply
> immediately an not to all platforms.
> But scale of applications grow. While engines and tools improve, larger
> content is getting developed, and it is already hitting that bottleneck.
> So solutions on this topic - are important to enable larger content for
> the
> web, not just small mini-games/demos.
>
> *#1: Nonblocking compile signaling (@khronos-wg)*
>> Could we introduce a new getProgramParameter variable like
>> gl.COMPILE_FINISHED or similar to produce nonblocking compile signaling?
>> (there was some discussion of this in another thread). As there's no
>> native
>> support AFAIK, how do we avoid stalling the GPU process when checking
>> the
>> gl.COMPILE_STATUS and therefore prevent other shaders from continued
>> compilation?
>
> I think, that would be great to have such check, and it feels like would
> be
> the easiest think to implement in short time.
>
> This would allow engines and tools to utilise this on platforms that
> provide such functionality. It will allow us to have non-blocking loading,
> as well as non-blocking shader compilation in runtime, with simply async
> results of compilation. Of course this shall be a user decision to choose
> behaviour, and tools can provide so.
> Although this would not reduce timing of compilation, only avoid blocking,
> asking developers to hide waiting somehow.
>
> Sync options fell like more robust and would reduce time spent dealing
> with
> shaders.
> Texture compression - is not mandatory, but powerful feature to be used.
> It
> would be amazing if there would be single path for compiled shaders, but
> due to nature of WebGL platform, that is not the case.
> So potentially platform based extension could be engaged.
>
> What important to remember, as with native tools, people don't write
> engines anymore to make 3D apps. They pick the tool that helps them to get
> to the target in efficient way. Back in 200s it wasn't a case. But today,
> same happening to WebGL platform - there is a urge for tools, and they are
> being developed, providing people with headache-free experience, allowing
> them to develop their apps, rather than struggle with the platform.
>
> It does not feel that binary shaders is a complex feature. All background
> workings of WebGL already has functionality to deal with it I'm sure. It
> is
> only about exposing such functionality using extensions notation based on
> platforms.
> That would provide block-free shaders, eliminating large JS stalls
> especially on mobile. On desktop those stalls renders related tabs as
> frozen, with pretty bad experience, that is comparable to if downloading
> images on websites would block JS thread :)
>
> Kind Regards,
> Max
>
> On 14 November 2016 at 13:52, Mr F <arthur@playcanvas.com> wrote:
>
>> >I don't think DX shader blobs are an acceptable solution at all, in any
>> scenario, for anybody.
>> Can you explain why? For me compiling shaders in a browser sounds no
>> better than compressing textures to GPU formats on-the-fly, which would
>> be
>> ridiculous. Both tasks are:
>> - Slow.
>> - Platform-dependent.
>> Compressed texture extensions are a great solution. Not every developer
>> uses them, and you have to use different 3rd party compressors to output
>> the right binary, but it's still pretty beneficial.
>> Fixing all the problems you mentioned involves a lot of time and people,
>> and it still will result in more overhead, comparing to directly passing
>> a
>> platform-dependent compiled shader right to the driver.
>>
>> On 14 November 2016 at 15:45, Florian Bösch <pyalot@gmail.com> wrote:
>>
>>> On Sat, Nov 12, 2016 at 9:54 PM, Mr F <arthur@playcanvas.com> wrote:
>>>
>>>> As there is more and more WebGL content around, shader compilation
>>>> times
>>>> start to annoy a lot (just go on shadertoy and see your tab hang for
>>>> half a
>>>> minute).
>>>>
>>> This is a known and not trivial problem. Slow shader compilation can
>>> stem
>>> from several factors:
>>>
>>>    1. ANGLE translation/unrolling, depending on circumstance it's
>>>    possible to hit slow paths in the ANGLE translation, though that's
>>> been
>>>    getting better.
>>>    2. Slow GL driver compilation, essentially unfixable.
>>>    3. HLSL compilation, usually better on newer versions of DX, except
>>>    edge cases.
>>>
>>>
>>>
>>>> Browsers try to cache already compiled shaders, but that's it, it's
>>>> hardly a solution IMHO.
>>>>
>>> No it isn't.
>>>
>>>
>>>> E.g. if you're on ANGLE, you can have a DX11 shader blob extension.
>>>> Just
>>>> feed it the binary, and it works. I don't think there's a security
>>>> concern
>>>> with it... you can make browsers compile almost any shader anyway, and
>>>> they
>>>> can't mess with unrelated user data.
>>>>
>>> I don't think DX shader blobs are an acceptable solution at all, in any
>>> scenario, for anybody.
>>>
>>>
>>> On Mon, Nov 14, 2016 at 12:49 PM, Maksims Mihejevs <max@playcanvas.com>
>>> wrote:
>>>
>>>> Worth mentioning this topic on that:
>>>> http://toji.github.io/shader-perf/ by Brandon Jones.
>>>>
>>> Asynchronous compilation doesn't fix that at the time it's checked, it
>>> will block.
>>>
>>>
>>>
>>>> We already doing something similar, where we access parameters in
>>>> delayed manner. This only works for assets loading scenarios, and does
>>>> not
>>>> work for runtime recompilation.
>>>> And even for loading scenario, this still takes ages, and in many
>>>> cases
>>>> shader compilation on medium devices can take more than half of
>>>> loading
>>>> times in extreme cases, that are seconds, for relatively simple demo
>>>> cases.
>>>>
>>>
>>> These problems have to be solved:
>>>
>>>    1. How to signal shader compile completion without blocking
>>>    (currently not possible)?
>>>    2. How to improve shader compilation speed drastically?
>>>       1. How to make sure ANGLE shader translation isn't compounding
>>> the
>>>       problem?
>>>       2. How to improve slow GL driver compilation?
>>>       3. How to avoid HLSL slow compiles?
>>>
>>> A relatively straightforward answer to this problem would be "Let's do
>>> WebVulkan", however we're a far way away from ever doing that. In the
>>> meantime, a solution would be nice.
>>>
>>>
>>>
>>> *#1: Nonblocking compile signaling (@khronos-wg)*
>>>
>>> Could we introduce a new getProgramParameter variable like
>>> gl.COMPILE_FINISHED or similar to produce nonblocking compile
>>> signaling?
>>> (there was some discussion of this in another thread). As there's no
>>> native
>>> support AFAIK, how do we avoid stalling the GPU process when checking
>>> the
>>> gl.COMPILE_STATUS and therefore prevent other shaders from continued
>>> compilation?
>>>
>>> *#2.1 Slow ANGLE translate phase (@google, @mozilla, @apple)*
>>>
>>> I'm not sure this is still a problem (though it might be with outdated
>>> ANGLE implementations), does this need more thorough testing? How to
>>> test
>>> this in isolation?
>>>
>>> *#2.2 Slow GL driver compiles (@nvidia, @amd, @intel)*
>>>
>>> There might not be anything that can be done to solve this (short of
>>> rewriting everybodies drivers). Is there some way to reduce compile
>>> times?
>>>
>>> *#2.3 Slow HLSL compiles (@microsoft)*
>>>
>>> This used to be a big problem in DX9 and apparently it got a bit better
>>> on DX11, though ANGLE based WebGL implementations still hit that
>>> problem
>>> occasionally, whereas it seems to me that Edge does less of that. Could
>>> you
>>> confirm that? If so, how does Edge avoid long compile times where ANGLE
>>> based HLSL backed still have it? Also how could HLSL compile times be
>>> improved in general?
>>>
>>
>>
>


 -- Steve


-----------------------------------------------------------
You are currently subscribed to public_webgl@khronos.org.
To unsubscribe, send an email to majordomo@khronos.org with
the following command in the body of your email:
unsubscribe public_webgl
-----------------------------------------------------------