To be thorough, we could verify every line of every algorithm in ES5 is indeed hit by one or more test cases.
As such an effort would require a tremendous test effort, it's debatable whether our limited resources should be spent on this. Perhaps instead it's enough to verify we have coverage for each and every 'section' (e.g., 11.1.1) of ES5.1.
As an extremely wild estimate, I'd imagine a couple of man-months of time would be needed to rigorously verify algorithm coverage versus several days for simple 'section' coverage.
Another approach could be to test coverage of the test suite against ES5.1 implementations. This would require access to the implementation code.
But SpiderMonkey is open source, so is the default WebKit JS engine, so is BESEN (http://besen.sourceforge.net/ not sure there exists coverage tools for Object Pascal though).
Maybe that it wouldn't be that hard to obtain cooperation of closed-source implementors to provide feedback on what aspects of their implementation isn't covered by the test suite and how in could be improved to do so.
I think that if several implementations passes all tests and the test suite covers entirely all implementations, it should mean with decent probability that the test suite is complete is very high.
I'm aware that there is some flaw in my point, but there are several ES5 implementations out there. After checking on Wikipedia, the most popular are written in C/C++. For such programs, there are tools such as gcov (http://gcc.gnu.org/onlinedocs/gcc/Gcov.html) which test coverage. Let's just test the different implementations against such tools, see for each what part isn't covered and see if the holes can be covered by other ES tests or if the hole is implementation specific.
I've been thinking more of my approach.
One of the issue with the spec is that it isn't in any formal language. As you note, it makes it almost impossible to review each step of every algorithm and note on the spec to make sure the test suite covers everything.
On the other hands, ES5.1 implementations are written in a formal language (C, C++ most of the time). As I said earlier, there are tools that can test program coverage against a test suite.
The only remaining issue I see with testing implementation coverage instead of the spec itself is that implementation could be incomplete. We could consider this issue as minor because:
- All implementors (especially web browsers) are "motivated" in providing full implementation to their users.
- The more implementation we test coverage on, the less likely they are to all be incomplete toward the exact same aspect of the spec.
I have never used coverage tools at a such scale, but they might also be able to provide information on the test suite, like if some test always test the same things, maybe they're duplicated. Implementors are going to check test failures, so these will be reviewed, but no one is going to manually review test that passes. The coverage tools could help to provide information on these.
Filed https://bugzilla.mozilla.org/show_bug.cgi?id=641214 in order to see what can be done to test Mozilla JS engine against test262 test suite.
*** Bug 55 has been marked as a duplicate of this bug. ***