[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Public WebGL] WebGL performance regression tests!











On Thu, Mar 29, 2012 at 4:03 PM, Benoit Jacob <bjacob@mozilla.com> wrote:
>
>
> ----- Original Message -----
>> On Thu, Mar 29, 2012 at 3:27 PM, Ilja Friedel <ihf@google.com> wrote:
>> > On Thu, Mar 29, 2012 at 2:21 PM, Benoit Jacob <bjacob@mozilla.com>
>> > wrote:
>> >>
>> >>
>> >> For what it's worth, the performance regression tests are now much
>> >> expanded, covering in particular the dreaded texture format
>> >> conversions.
>> >>
>> >> Still same URL:
>> >>
>> >> http://hg.mozilla.org/users/bjacob_mozilla.com/webgl-perf-tests/raw-file/tip/webgl-performance-tests.html
>> >
>> >
>> > I second the very nice. To run this in practice in a regression
>> > suite one
>> > would need an easy way to adjust the work done for each individual
>> > test to
>> > say 50ms or 100ms. Right now on my atom test runtimes are between
>> > 8ms and
>> > 517ms. 8ms is awfully hard to distinguish from zero and computing
>> > any sum
>> > (or even geometric mean) is dodgy. Maybe allow for some kind of
>> > configuration/ parameterization of the number of loops running for
>> > each
>> > test?
>>
>> Good point. The V8 benchmark suite generally runs each benchmark for
>> a
>> fixed period of time, say 2 seconds, and measures the number of
>> iterations of the test that can be performed in that time, rather
>> than
>> using a fixed number of iterations. This seems to be a pretty robust
>> methodology.
>
> The performance tests currently run until both conditions are met: they've either done 10 iterations, and run for at least 300 ms.
>
> See in WebGLPerformanceTest.js:
>
> WebGLPerformanceTest.prototype.hasRecordedEnoughFrames = function(time) {
>  // stop when we have recorded at least 10 frames and run for at least 300 ms
>  return this.timings.length > 10 &&
>         time - this.startTime > 300;
> }
>
> The final result is then computed as the median result among all iterations/frames.
>
> So when you get "8 ms", it doesn't mean that the test stopped after 8 ms. It means that 8 ms is the median frame time.

In understand. And that means that we are mostly there! What I would like to see is (at least) 2 significant digits so one can pick up a 1 percent change in performance. In the run below (on an low end atom) many numbers are 20ms or below. This means that a change in performance is only detected when times changes more than 1ms/20ms or 5 percent. This problem obviously gets worse with faster hardware.
Two problems here:
 - I am not aware of timing mechanisms exposed to _javascript_, that would have better than millisecond precision. There is some consensus at Mozilla to make that happen, but it hasn't happened yet AFAIK.
 - Most test results have more discrepancies across runs, than this 1ms precision, anyway. So in most cases (including when the result is < 10 ms) we often have a bigger problem here anyway.

Both can potentially be addressed by adding more workload to get higher timings and reduce noise.


Finally perf numbers are often aggregated into a single number by summing them up (bad!) or using a geometric mean (better, but problematic when we get many results). Either way I am worried what happens if the reported numbers start approaching zero.
I was wondering if we would want such a synthetic measure. Let's give them a chance to prove their usefulness ;-)
Done. I'm excited because these averages, especially the geometric one, seem very stable across runs.

Benoit


Benoit

Ilja.


WebGL Performance Regression Test Results

Keep in mind that these tests are not realistic workloads. These are not benchmarks aiming to compare browser or GPU performance. These are only useful to catch performance regressions in a given browser and system.

Test page ResultDescription
clear-10x-default.html 484 msClear with defaults, repeated 10Ã, size 1024Ã1024
clear-10x-nondefault-constant-color.html 332 ms Clear with constant non-default color, repeated 10Ã, size 1024Ã1024
clear-default.html 51 msClear with defaults, size 1024Ã1024
clear-default-preserveDrawingBuffer.html 59 ms Clear with defaults, with preserveDrawingBuffer, size 1024Ã1024
clear-nondefault-constant-color.html 64 msClear with constant non-default color, size 1024Ã1024
clear-nondefault-constant-color-preserveDrawingBuffer.html 56 ms Clear with constant non-default color, with preserveDrawingBuffer, size 1024Ã1024
clear-varying-color-and-readPixels.html 159 msClear with varying color and readPixels, size 1024Ã1024
clear-varying-color-and-readPixels-out-of-bounds.html 182 ms Clear with varying color and readPixels out-of-bounds, size 1024Ã1024
clear-varying-color.html 52 msClear with varying color, size 1024Ã1024
clear-varying-color-preserveDrawingBuffer.html 51 ms Clear with varying color, with preserveDrawingBuffer, size 1024Ã1024
convert-Canvas-to-a8.html 14 msTexture conversion, from 1024Ã1024 Canvas to A8 unpremult.
convert-Canvas-to-a8-premultiplied.html 14 msTexture conversion, from 1024Ã1024 Canvas to A8 premult.
convert-Canvas-to-l8.html 76 ms Texture conversion, from 1024Ã1024 Canvas to L8 unpremult.
convert-Canvas-to-l8-premultiplied.html 15 ms Texture conversion, from 1024Ã1024 Canvas to L8 premult.
convert-Canvas-to-rgb565.html 109 msTexture conversion, from 1024Ã1024 Canvas to RGB565 unpremult.
convert-Canvas-to-rgb565-premultiplied.html 24 ms Texture conversion, from 1024Ã1024 Canvas to RGB565 premult.
convert-Canvas-to-rgb888.html 107 msTexture conversion, from 1024Ã1024 Canvas to RGB888 unpremult.
convert-Canvas-to-rgb888-premultiplied.html 22 ms Texture conversion, from 1024Ã1024 Canvas to RGB888 premult.
convert-Canvas-to-rgba4444.html 115 msTexture conversion, from 1024Ã1024 Canvas to RGBA4444 unpremult.
convert-Canvas-to-rgba4444-premultiplied.html 21 ms Texture conversion, from 1024Ã1024 Canvas to RGBA4444 premult.
convert-Canvas-to-rgba8888.html 107 msTexture conversion, from 1024Ã1024 Canvas to RGBA8888 unpremult.
convert-Canvas-to-rgba8888-premultiplied.html 18 ms Texture conversion, from 1024Ã1024 Canvas to RGBA8888 premult.
convert-Canvas-to-rgb-float.html SkippedTexture conversion, from 1024Ã1024 Canvas to RGBA/float unpremult.
convert-Canvas-to-rgb-float-premultiplied.html Skipped Texture conversion, from 1024Ã1024 Canvas to RGBA/float premult.
convert-ImageData-to-a8.html 7 msTexture conversion, from 1024Ã1024 ImageData to A8 unpremult.
convert-ImageData-to-a8-premultiplied.html 8 ms Texture conversion, from 1024Ã1024 ImageData to A8 premult.
convert-ImageData-to-l8.html 8 msTexture conversion, from 1024Ã1024 ImageData to L8 unpremult.
convert-ImageData-to-l8-premultiplied.html 47 ms Texture conversion, from 1024Ã1024 ImageData to L8 premult.
convert-ImageData-to-rgb565.html 17 msTexture conversion, from 1024Ã1024 ImageData to RGB565 unpremult.
convert-ImageData-to-rgb565-premultiplied.html 82 ms Texture conversion, from 1024Ã1024 ImageData to RGB565 premult.
convert-ImageData-to-rgb888.html 15 msTexture conversion, from 1024Ã1024 ImageData to RGB888 unpremult.
convert-ImageData-to-rgb888-premultiplied.html 82 ms Texture conversion, from 1024Ã1024 ImageData to RGB888 premult.
convert-ImageData-to-rgba4444.html 16 msTexture conversion, from 1024Ã1024 ImageData to RGBA4444 unpremult.
convert-ImageData-to-rgba4444-premultiplied.html 87 ms Texture conversion, from 1024Ã1024 ImageData to RGBA4444 premult.
convert-ImageData-to-rgba8888.html 14 msTexture conversion, from 1024Ã1024 ImageData to RGBA8888 unpremult.
convert-ImageData-to-rgba8888-premultiplied.html 84 ms Texture conversion, from 1024Ã1024 ImageData to RGBA8888 premult.
convert-ImageData-to-rgb-float.html SkippedTexture conversion, from 1024Ã1024 ImageData to RGBA/float unpremult.
convert-ImageData-to-rgb-float-premultiplied.html Skipped Texture conversion, from 1024Ã1024 ImageData to RGBA/float premult.
convert-Image-to-a8.html 11 msTexture conversion, from 1024Ã1024 Image to A8 unpremult.
convert-Image-to-a8-premultiplied.html 12 msTexture conversion, from 1024Ã1024 Image to A8 premult.
convert-Image-to-l8.html 16 ms Texture conversion, from 1024Ã1024 Image to L8 unpremult.
convert-Image-to-l8-premultiplied.html 15 ms Texture conversion, from 1024Ã1024 Image to L8 premult.
convert-Image-to-rgb565.html 21 msTexture conversion, from 1024Ã1024 Image to RGB565 unpremult.
convert-Image-to-rgb565-premultiplied.html 20 ms Texture conversion, from 1024Ã1024 Image to RGB565 premult.
convert-Image-to-rgb888.html 21 msTexture conversion, from 1024Ã1024 Image to RGB888 unpremult.
convert-Image-to-rgb888-premultiplied.html 22 ms Texture conversion, from 1024Ã1024 Image to RGB888 premult.
convert-Image-to-rgba4444.html 20 msTexture conversion, from 1024Ã1024 Image to RGBA4444 unpremult.
convert-Image-to-rgba4444-premultiplied.html 21 ms Texture conversion, from 1024Ã1024 Image to RGBA4444 premult.
convert-Image-to-rgba8888.html 19 msTexture conversion, from 1024Ã1024 Image to RGBA8888 unpremult.
convert-Image-to-rgba8888-premultiplied.html 17 ms Texture conversion, from 1024Ã1024 Image to RGBA8888 premult.
convert-Image-to-rgb-float.html SkippedTexture conversion, from 1024Ã1024 Image to RGBA/float unpremult.
convert-Image-to-rgb-float-premultiplied.html Skipped Texture conversion, from 1024Ã1024 Image to RGBA/float premult.
decrement-dimensions.html 183 msDecrement canvas dimensions, size 1024Ã1024
do-nothing.html 18 msDo nothing, size 1024Ã1024
do-nothing-preserveDrawingBuffer.html 17 msDo nothing, with preserveDrawingBuffer, size 1024Ã1024
increment-dimensions.html 70 msIncrement canvas dimensions, size 1024Ã1024
texImage2D-null.html 1 ms texImage2D with null, repeated 10Ã
texImage2D-TypedArray.html 87 ms texImage2D with TypedArray, repeated 10Ã
texSubImage2D-TypedArray.html 84 ms texSubImage2D with TypedArray, repeated 10Ã