[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Public WebGL] WebGL Conformance Testing Harness First Attempt



On Feb 8, 2010, at 2:58 PM, Gregg Tavares wrote:

> I put together a testing harness for WebGL.

This all looks very good. My initial comment is that we should have the harness check to see if your browser has WebGL support at all, so you don't waste your time getting all red tests!

> 
> My goals were as follows:
> 
> 	• Tests should exist in separate html files that can be run on their own without the harness
> 	• Harness should be as light weight as possible so it can easily work with any testing framework
> 	• Harness should show some info about each test that fails (vs just a count of pass/fail)

I like how you were able to integrate well with the WebKit test harness. That will make it easier to port tests over.

> 
> With those goals in mind here's a first attempt at a harness that meets those goals.
> 
> https://cvs.khronos.org/svn/repos/registry/trunk/public/webgl/sdk/tests/webgl-conformance-tests.html
> 
> I included all the tests that were already there and I added tests for the issues I felt most implementations were likely to still have problems with. Most of those problems have to do with the fact that all implementations are running on top of desktop OpenGL and are not rejecting enums and other parameters that are not valid for OpenGL ES 2.0.
> 
> The most important issue is with the differences with shaders. To get desktop OpenGL to behave like the OpenGL ES 2.0 spec will require implementations to both pre-process and parse the shaders themselves and munge them to remove stuff that works in GLES2 but not in DesktopGL, to enforce things that GLES2 requires but DesktopGL does not (precision specifications) and to disallow DesktopGL stuff that is not allowed in GLES2 (gl_ProjectionMatrix, gl_ClipVertex, ...).
> 
> It's easy to add new tests. If you use the framework there it just works.  You need to add your test to sdk/tests/fast/00_test_list.txt  or you can add to sdk/tests/test_list.txt.  The harness reads starts reading the later and adds the pages listed inside. If it sees another .txt file it will parse those as well for more tests. 

Maybe it would be better to avoid the test_list.txt file and rather have a skip list like in WebKit. That still gives us the ability to omit tests, but easier to add them and not have to remember to add them to a separate list.

> 
> Otherwise, if you prefer a different framework, to make a test page work with the harness the test page needs to call 
> 
>     window.parent.webglTestHarness.reportResults(success, msg);
> 
> for each test passing true or false for success and a message about the test. 
> 
> Once all the tests on the page have run the page must call
> 
>     window.parent.webglTestHarness.notifyFinished();
> 
> If notifyFinished is not called in a reasonable amount of time the harness will assume the page timed out and mark it as failed. 
> 
> Both of these calls are easily wrapped with checks for their existence so that if the page is not run from the harness nothing happens. As an example, to make all the tests that were already checked in work all I had to do was add these two functions to sdk/tests/resources/js-test-pre.js
> 
> function reportTestResultsToHarness(success, msg) {
>   if (window.parent.webglTestHarness) {
>     window.parent.webglTestHarness.reportResults(success, msg);
>   }
> }
> 
> function notifyFinishedToHarness() {
>   if (window.parent.webglTestHarness) {
>     window.parent.webglTestHarness.notifyFinished();
>   }
> }
> 
> I then updated testPassed and testFailed as follows
> 
> function testPassed(msg)
> {
>     reportTestResultsToHarness(true, msg);
>     debug('<span><span class="pass">PASS</span> ' + escapeHTML(msg) + '</span>');
> 
> 
> 
> }
> 
> 
> function testFailed(msg)
> {
>     reportTestResultsToHarness(false, msg);
>     debug('<span><span class="fail">FAIL</span> ' + escapeHTML(msg) + '</span>');
> 
> 
> 
> }
> 
> And I updated sdk/tests/resources/js-test-post.js as follows
> 
> shouldBeTrue("successfullyParsed");
> debug('<br /><span class="pass">TEST COMPLETE</span>');
> notifyFinishedToHarness();
> 
> I didn't make a lot of effort to style the harness. Everything having to do with UI and display is in sdk/tests/webgl-conformance-tests.html so if someone wants to make it prettier be my guest.

Yeah, making it pretty can come later.

> Otherwise it mostly seems to work. Some implementations need to synthesize GL errors instead of throwing exceptions.
> 
> Of course the tests represent my understanding of the spec. Please test your implementations and bring up any issues you find.
> 

All looks good. Next I'd like to see if we can integrate some sort of pixel tests. I think we can come up with reasonable results with at least simple static scenes.

What is Ilmari's status? Can we integrate his tests into this harness?

-----
~Chris
cmarrin@apple.com





-----------------------------------------------------------
You are currently subscribe to public_webgl@khronos.org.
To unsubscribe, send an email to majordomo@khronos.org with
the following command in the body of your email: