Khronos Public Bugzilla
Bug 793 - Bug to keep track of the WebCL conformance test effort
Bug to keep track of the WebCL conformance test effort
Status: NEW
Product: WebCL
Classification: Unclassified
Component: Conformance Tests
1.0
All All
: P3 normal
: ---
Assigned To: WebCL Mailing List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2013-02-08 18:02 PST by Tasneem Brutch
Modified: 2014-01-30 10:26 PST (History)
8 users (show)

See Also:


Attachments
Conformance Test Review & Status (47.96 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
2013-02-13 18:49 PST, steven eliuk
Details
Conformance Test Review & Status (April/24) (21.45 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
2013-04-24 21:33 PDT, Antonio Gomes
Details
Conformance Test Review & Status.(May/13). (21.87 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
2013-05-13 03:34 PDT, Sharath Kamath
Details
Conformance Test Review & Status (Aug/01) (66.48 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
2013-08-01 10:46 PDT, Antonio Gomes
Details
Conformance Test Review & Status (67.50 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
2013-08-12 17:12 PDT, steven eliuk
Details
Conformance Test Review & Status (134.74 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
2013-10-16 06:04 PDT, Sharath Kamath
Details
Conformance Test Review & Status (137.19 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
2013-10-31 04:51 PDT, Shilpa shri
Details
Conformance Test Review & Status (406.39 KB, application/vnd.ms-excel)
2013-11-05 03:54 PST, Shilpa shri
Details
Conformance Spreadsheet 11-05-13 (418.50 KB, application/octet-stream)
2013-11-05 13:08 PST, steven eliuk
Details
Conformance Test Review & Status (307.89 KB, application/vnd.ms-excel)
2013-12-13 03:47 PST, Sharath Kamath
Details
Updated WebCL Conformance Test Status as of 13 Dec, 2013. (326.00 KB, application/vnd.ms-excel)
2013-12-13 09:19 PST, Tasneem Brutch
Details
WebCL Conformance Test Status as of 30 Jan, 2014. (295.39 KB, application/vnd.ms-excel)
2014-01-30 03:56 PST, Shilpa shri
Details
WebCL Conformance Test Status as of 30 Jan, 2014. (289.00 KB, application/vnd.ms-excel)
2014-01-30 05:12 PST, Sharath Kamath
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tasneem Brutch 2013-02-08 18:02:43 PST
Opening a bug for tracking the following:
(1) Lists of WebCL conformance tests needed.
(2) Conformance tests completed.
(3) Exit criteria for completion of conformance test effort.
Comment 1 steven eliuk 2013-02-13 18:49:53 PST
Created attachment 135 [details]
Conformance Test Review & Status

The current conformance tests suite contains approx. 380-400 unique tests. There are a number of issues identified, such as:
-many of the tests do not reflect the current WebCL-WD specification,
-there are many redundant tests,
-incomplete tests,
-tests that are no longer applicable.

From my analysis the conformance suite is only approx. 40-percent complete.

Therefore, we need a plan of action to complete the conformance test suite.

Please see the attached spreadsheet and update as needed,
Comment 2 Tasneem Brutch 2013-02-13 19:01:21 PST
The current set of conformance tests contributed by Samsung can be accessed from:
https://github.com/KhronosGroup/WebCL-conformance
Comment 3 Antonio Gomes 2013-02-14 06:45:18 PST
(In reply to comment #1)
> Created attachment 135 [details]
> Conformance Test Review & Status
> 
> The current conformance tests suite contains approx. 380-400 unique tests.
> There are a number of issues identified, such as:
> -many of the tests do not reflect the current WebCL-WD specification,
> -there are many redundant tests,
> -incomplete tests,
> -tests that are no longer applicable.
> 
> From my analysis the conformance suite is only approx. 40-percent complete.
> 
> Therefore, we need a plan of action to complete the conformance test suite.
> 
> Please see the attached spreadsheet and update as needed,

I do not think we are at 40%, personally. If all the above is fixed, it might go down to about 20%.

- Many tests differ one line from one to another. These could get collapsed into one, maybe.
- Many tests duplicate lots of code as well, specially on error handling. I think this is must-address issue.
Comment 4 Tomi Aarnio 2013-02-14 07:09:08 PST
(In reply to comment #3)
> - Many tests duplicate lots of code as well, specially on error handling. I
> think this is must-address issue.

I agree. There's a staggering amount of redundant/duplicated code in the test suite as it stands. Any changes that we need to make w.r.t. context creation, kernel loading, error handling, presentation of results, etc., must now be replicated into dozens or hundreds of different HTML files. That's not sustainable.

We need a solid framework that separates commonly used code from code that is specific to each test case. Also, it would be a good idea to have kernel code and host code in separate files (.cl and .js, respectively), instead of cramming everything inside html.
Comment 5 Tomi Aarnio 2013-02-14 08:28:16 PST
I heartily recommend that we port the conformance tests to the Jasmine framework:

  http://pivotal.github.com/jasmine/

Jasmine is extremely elegant and easy to learn. In less than one hour, I was able to write some basic test cases for WebCL, including context creation and platform discovery -- and I hadn't even heard about Jasmine before! Check it out:

  http://webcl.nokiaresearch.com/tests/jasmine/SpecRunner.html

And again, I wrote that in less than one hour from scratch, and had so much fun that I missed the first half of our conference call! :-)
Comment 6 Siba Prasad Samal 2013-02-19 06:16:39 PST
I agree with the view of pushing common codes mainly "kernel codes" into separate files/folder. But the presentation of results are similar to WebGL style which allows selection of the parts of tests to be run.
Comment 7 Antonio Gomes 2013-03-12 11:56:22 PDT
> I agree. There's a staggering amount of redundant/duplicated code in the
> test suite as it stands. Any changes that we need to make w.r.t. context
> creation, kernel loading, error handling, presentation of results, etc.,
> must now be replicated into dozens or hundreds of different HTML files.
> That's not sustainable.

That is being worked on.
 
> We need a solid framework that separates commonly used code from code that
> is specific to each test case. Also, it would be a good idea to have kernel
> code and host code in separate files (.cl and .js, respectively), instead of
> cramming everything inside html.

Ditto.

Other clean ups are also being performed as well.

Once we reach a level of quality regarding what is currently there and validate with the group, we will proceed with adding more tests.

The spreadsheet attached by Steven will be re-uploaded here in a weekly basis, after the clean up is done. It will comprise the update on the APIs currently being tested, and the progresses per week.

Let me know if you have better suggestions, please.
Comment 8 Antonio Gomes 2013-04-24 21:33:33 PDT
Created attachment 140 [details]
Conformance Test Review & Status (April/24)

Updated spreadsheet, with the latest status about the existing tests.
Comment 9 Sharath Kamath 2013-05-13 03:31:40 PDT
The updated conformance test suite is pushed to Khronos github.
Please find the updated status on in the xls : Conformance Test Review & Status.(May/13).
Comment 10 Sharath Kamath 2013-05-13 03:34:08 PDT
Created attachment 141 [details]
Conformance Test Review & Status.(May/13).
Comment 11 Tomi Aarnio 2013-05-16 09:34:20 PDT
I took a quick look at the new set of conformance tests, and it looks much better than last time (which was some time last year). I'm particularly happy to see the WebCL boilerplate code separated into a utilities class.

Looking at the code of the individual test cases (I didn't try to run any), a couple of things caught my eye:

 1. When testing exceptions, you're checking that the correct type of exception is thrown (e.g., "INVALID_VALUE"). I think it would suffice to check whether *any* exception is thrown, regardless of type. I'm concerned that being too strict about this is going to make all existing OpenCL drivers to fail WebCL conformance.

 2. The tests are apparently run on only one device, which is the default device selected by the WebCL implementation -- correct? Given that multi-device systems are pretty common, it might be a good idea to let the user select which device to use.
Comment 12 Antonio Gomes 2013-05-16 09:47:04 PDT
(In reply to comment #11)
>  2. The tests are apparently run on only one device, which is the default
> device selected by the WebCL implementation -- correct? Given that
> multi-device systems are pretty common, it might be a good idea to let the
> user select which device to use.

Right, that is a known issue. We have it in our roadmap that it should be possible to run tests on any/all OpenCL-enabled device available.

In the redesigned home page, one will be able to toggle on/off specific devices to run tests on, or all (default).

Same apply for platforms.
Comment 13 Antonio Gomes 2013-07-15 12:39:36 PDT
(In reply to comment #11)

>  1. When testing exceptions, you're checking that the correct type of
> exception is thrown (e.g., "INVALID_VALUE"). I think it would suffice to
> check whether *any* exception is thrown, regardless of type. I'm concerned
> that being too strict about this is going to make all existing OpenCL
> drivers to fail WebCL conformance.

To follow up on this, it has been implemented, and will be available at the next code drop from Samsung wrt conformance test suite.
Comment 14 Antonio Gomes 2013-08-01 10:46:58 PDT
Created attachment 161 [details]
Conformance Test Review & Status (Aug/01)

New spreadsheet version that reflects to the code available on Khronos GitHub branch https://github.com/KhronosGroup/WebCL-conformance/tree/For_Review_08012013 .
Comment 15 steven eliuk 2013-08-12 17:12:43 PDT
Created attachment 162 [details]
Conformance Test Review & Status


Updated spreadsheet version that reflects to the code available on Khronos GitHub branch https://github.com/KhronosGroup/WebCL-conformance/tree/For_Review_08012013 

The new spreadsheet contains a column for the 'reviewers' name and affiliation. 

Please select a block and update the spreasheet, 

0) Rows [3 – 6], --Steven Eliuk, Samsung --
1) Rows [7 – 10],  --Steven Eliuk, Samsung --
2) Rows [11 – 14],
3) Rows [15 – 18],
4) Rows [19 – 22],
5) Rows [23 – 26],
6) Rows [27 – 30], --Antonio Gomes, Samsung --
7) Rows [31 – 35],
8) Rows [38 – 42],
9) Rows [45 – 48],
10) Rows [49 – 53],
11) Rows [56 – 59],
12) Rows [62 – 67],
13) Rows [70 – 73],
14) Rows [76 – 80],
15) Rows [95 – 96] & [99 – 100].
Comment 16 steven eliuk 2013-08-12 17:20:14 PDT
0) Rows [3 – 6], --Steven Eliuk, Samsung --
1) Rows [7 – 10],  --Steven Eliuk, Samsung --
2) Rows [11 – 14], --Tentatively, Jeff Gilbert, Mozilla--
3) Rows [15 – 18], --Tentatively, Jeff Gilbert, Mozilla--
4) Rows [19 – 22], --Tentatively, Jeff Gilbert, Mozilla--
5) Rows [23 – 26], --Tentatively, Jeff Gilbert, Mozilla--
6) Rows [27 – 30], --Antonio Gomes, Samsung --
7) Rows [31 – 35],
8) Rows [38 – 42],
9) Rows [45 – 48],
10) Rows [49 – 53],
11) Rows [56 – 59],
12) Rows [62 – 67],
13) Rows [70 – 73],
14) Rows [76 – 80],
15) Rows [95 – 96] & [99 – 100].
Comment 17 steven eliuk 2013-08-19 11:57:37 PDT
0) Rows [3 – 6], --Steven Eliuk, Samsung --
1) Rows [7 – 10],  --Steven Eliuk, Samsung --
2) Rows [11 – 14], --Tentatively, Jeff Gilbert, Mozilla--
3) Rows [15 – 18], --Tentatively, Jeff Gilbert, Mozilla--
4) Rows [19 – 22], --Tentatively, Jeff Gilbert, Mozilla--
5) Rows [23 – 26], --Tentatively, Jeff Gilbert, Mozilla--
6) Rows [27 – 30], --Antonio Gomes, Samsung --
7) Rows [31 – 35],
8) Rows [38 – 42],
9) Rows [45 – 48],
10) Rows [49 – 53],
11) Rows [56 – 59],
12) Rows [62 – 67], --Anthony Liot-- 
13) Rows [70 – 73],
14) Rows [76 – 80],
15) Rows [95 – 96] & [99 – 100].
Comment 18 fabien cellier 2013-08-22 06:19:44 PDT
 0) Rows [3 – 6], --Steven Eliuk, Samsung --
 1) Rows [7 – 10],  --Steven Eliuk, Samsung --
 2) Rows [11 – 14], --Tentatively, Jeff Gilbert, Mozilla--
 3) Rows [15 – 18], --Tentatively, Jeff Gilbert, Mozilla--
 4) Rows [19 – 22], --Tentatively, Jeff Gilbert, Mozilla--
 5) Rows [23 – 26], --Tentatively, Jeff Gilbert, Mozilla--
 6) Rows [27 – 30], --Antonio Gomes, Samsung --
 7) Rows [31 – 35],
 8) Rows [38 – 42],
 9) Rows [45 – 48],
 10) Rows [49 – 53],
 11) Rows [56 – 59],
 12) Rows [62 – 67], --Anthony Liot-- 
 13) Rows [70 – 73], 
 14) Rows [76 – 80],-- Fabien Cellier --
 15) Rows [95 – 96] & [99 – 100].
Comment 19 fabien cellier 2013-08-23 08:32:39 PDT
(In reply to comment #18)
>  0) Rows [3 – 6], --Steven Eliuk, Samsung --
>  1) Rows [7 – 10],  --Steven Eliuk, Samsung --
>  2) Rows [11 – 14], --Tentatively, Jeff Gilbert, Mozilla--
>  3) Rows [15 – 18], --Tentatively, Jeff Gilbert, Mozilla--
>  4) Rows [19 – 22], --Tentatively, Jeff Gilbert, Mozilla--
>  5) Rows [23 – 26], --Tentatively, Jeff Gilbert, Mozilla--
>  6) Rows [27 – 30], --Antonio Gomes, Samsung --
>  7) Rows [31 – 35],
>  8) Rows [38 – 42],
>  9) Rows [45 – 48],
>  10) Rows [49 – 53],
>  11) Rows [56 – 59],
>  12) Rows [62 – 67], --Anthony Liot-- 
>  13) Rows [70 – 73],-- Fabien Cellier --
>  14) Rows [76 – 80],-- Fabien Cellier --
>  15) Rows [95 – 96] & [99 – 100].
Comment 20 Sharath Kamath 2013-10-16 06:04:03 PDT
Created attachment 168 [details]
Conformance Test Review & Status

Updating the spreadsheet to reflect the test suite status in https://github.com/KhronosGroup/WebCL-conformance/tree/master_r10162013.
Comment 21 Shilpa shri 2013-10-31 04:51:43 PDT
Created attachment 173 [details]
Conformance Test Review & Status

Re-uploading the spreadsheet, replacing the corrupted spreadsheet.

Spreadsheet reflects the test suite status in
https://github.com/KhronosGroup/WebCL-conformance/tree/master_r10162013
Comment 22 Shilpa shri 2013-11-05 03:54:31 PST
Created attachment 174 [details]
Conformance Test Review & Status

Re-uploading the spreadsheet, replacing the corrupted spreadsheet.

Spreadsheet reflects the test suite status in
https://github.com/KhronosGroup/WebCL-conformance/tree/master_r10162013
Comment 23 steven eliuk 2013-11-05 13:08:25 PST
Created attachment 175 [details]
Conformance Spreadsheet 11-05-13

Updated Spreadsheet,
Comment 24 Sharath Kamath 2013-12-13 03:47:00 PST
Created attachment 178 [details]
Conformance Test Review & Status

Updating the spreadsheet to reflect the status of conformance test suite wrt https://github.com/KhronosGroup/WebCL-conformance/tree/master_r12132013 branch.

Also added a new column to reflect the status.
[green = complete, yellow = in progress, red = not started]
Comment 25 Tasneem Brutch 2013-12-13 09:19:17 PST
Created attachment 179 [details]
Updated WebCL Conformance Test Status as of 13 Dec, 2013.

Updated WebCL Conformance Test Status as of 13 Dec, 2013 is attached, with status update and color coding to indicate status.
Comment 26 Shilpa shri 2014-01-30 03:56:46 PST
Created attachment 180 [details]
WebCL Conformance Test Status as of 30 Jan, 2014.

WebCL Conformance Test Status as of 30 Jan, 2014.

Updated WebCL Conformance Test Status as of 30 Jan, 2014 is attached, with status update and color coding to indicate status.
Comment 27 Sharath Kamath 2014-01-30 05:12:25 PST
Created attachment 181 [details]
WebCL Conformance Test Status as of 30 Jan, 2014.

Attaching a non-encrypted copy of the xls.
Comment 28 David Ligon 2014-01-30 10:26:32 PST
I think I should not have to visit this bug to track status of conformance tests.  As the test suite is an on-going process this bug will never close, or the status information will cease if it does.  Please put tracking information in a more permanent location.