Created attachment 74
Additional tests for missing semantic rules
The following semantic rules are not tested by test262:
- 8.7.1 GetValue (V) - [[Get]], Step 6
- 8.7.2 PutValue (V, W) - [[Put]], Step 2.a
- 8.7.2 PutValue (V, W) - [[Put]], Step 2.b
- 8.7.2 PutValue (V, W) - [[Put]], Step 6.a & 6.b
- 8.7.2 PutValue (V, W) - [[Put]], Step 7.a
- 8.12.4 [[CanPut]] (P) - Step 8.a
- 10.2.1.1.3 SetMutableBinding (N,V,S) - Step 4
- 10.2.1.1.5 DeleteBinding (N) - Step 4 & 5
- 10.5 Declaration Binding Instantiation - Step 5.e.iii.1
- 10.5 Declaration Binding Instantiation - Step 5.e.iv, 1st condition is true
- 10.5 Declaration Binding Instantiation - Step 5.e.iv, 2nd condition is true
I manually wrote test programs for each cases, attached below, and running these tests against major browsers (Chrome, Firefox, Safari) revealed a couple of known and unknown bugs.
Known bugs:
https://code.google.com/p/v8/issues/detail?id=2243 (recently fixed)
=> 10.2.1.1.3 SetMutableBinding (N,V,S) - Step 4
https://bugzilla.mozilla.org/show_bug.cgi?id=779682 (not fixed yet)
=> 8.7.2 PutValue (V, W) - [[Put]], Step 7.a
Unknown bugs:
https://bugs.webkit.org/show_bug.cgi?id=138859 (confirmed)
=> 10.5 Declaration Binding Instantiation - Step 5.e.iv, 1st condition is true
=> 10.5 Declaration Binding Instantiation - Step 5.e.iv, 2nd condition is true
https://bugs.webkit.org/show_bug.cgi?id=138858
=> 10.2.1.1.3 SetMutableBinding (N,V,S) - Step 4
The uncovered semantic rules were found by systematically measuring test coverage of test262 using a formal JS semantics that I recently developed. The semantics is suitable for test coverage measurement, since it is executable and closely resembles ES5 document. I think that the semantics is helpful to solve the test262's test coverage problem, reported several times:
https://bugzilla.mozilla.org/show_bug.cgi?id=641214
https://github.com/rwaldron/tc39-notes/blob/master/es6/2014-09/sept-23.md#somehow-we-started-talking-about-test262
Note that, however, I measured only the statement coverage, not the branch coverage nor the condition coverage. Also, I measured only for the language core (i.e., Chapter 8-14), not for the standard built-in objects (i.e., Chapter 15). Thus, there may exist more missing cases. More comprehensive test coverage measurement is a future work.
For more details, please refer to the following page:
https://github.com/kframework/javascript-semantics/tree/master/test262-coverage
Thanks,
Daejun