I'm smoke testing and validating updates to our (SUSE) nodejs package using this test suite.
For that, I'm packaging the test suite into an rpm package. The package is rebuilt automatically when the nodejs package is updated (<plug>on all architectures and all distributions in the build service. That includes, if someone has interest, fedora, ubuntu and CentOS, not only SUSE. If someone has interest I can help enabling these, too. Oh, and it could easily be extended to other js runtimes, too.</plug>)
To increase the value of the test builds, I include the result of a manual reviewed run in the package and I compare to that, using goode olde diff -u.
Now the current log of the test suite is garbled if a failing test output doesn't end with a newline. In those cases the log will contain that last line of the failed test together with the name of the next test.
My little sed work around shows the issue nicely:
0 froh@byron:/tmp/test262 $ diff -u test262.log <(sed -e "s,\^\([^\n]\),^\n\1," test262.log )
--- test262.log 2013-08-14 15:40:33.141662499 +0200
+++ /dev/fd/63 2013-08-14 15:54:35.264134519 +0200
@@ -1981,7 +1981,8 @@
--- output ---
/tmp/test262-TEST.js:289: Test262 Error: Test case returned non-true value!
throw new Test262Error(message);
- ^ch11/11.2/11.2.3/11.2.3-3_4 passed in non-strict mode
+ ^
+ch11/11.2/11.2.3/11.2.3-3_4 passed in non-strict mode
ch11/11.2/11.2.3/11.2.3-3_5 passed in non-strict mode
ch11/11.2/11.2.3/11.2.3-3_6 passed in non-strict mode
ch11/11.2/11.2.3/11.2.3-3_7 passed in non-strict mode
@@ -4484,7 +4485,8 @@
--- output ---
/tmp/test262-TEST.js:289: Test262 Error: #1: JSON.parse confused by "__proto__"
throw new Test262Error(message);
- ^ch15/15.12/15.12.3/15.12.3-0-1 passed in non-strict mode
+ ^
+ch15/15.12/15.12.3/15.12.3-0-1 passed in non-strict mode
ch15/15.12/15.12.3/15.12.3-0-2 passed in non-strict mode
ch15/15.12/15.12.3/15.12.3-0-3 passed in non-strict mode
ch15/15.12/15.12.3/15.12.3-11-1 passed in non-strict mode
...
I'd appreciate a fix for this, probably in tools/packaging/test262.py:TestResult.ReportOutcome(), as my workaround only captures that one kind of borked output that currently is there.
As I don't know in which scripts you possibly use the log file, I didn't just create a patch.
thx :)
S.
PS: the tempnames can be handled fine with sed, like so:
sed -e "s,/tmp/test262-......\.js:,/tmp/test262-TEST.js:,"
Created attachment 68
add newlines after --- output --- in log files
ok, I realized that there is nicely obfuscated code duplication, writing to the screen uses 'print' statements, while writing to the log file writes to the log file, and these two are not consistent.
Furthermore, the log file is scattered with trailing whitespace on the lines, which greatly obfuscated the issue :D
anyhow, attached is a patch that fixes exactly one thing: add newlines after
--- output --- for failing test cases.
As of 2014-July-17, this is not present in test262.py.
I am willing to convert this patch into a github pull request.
ah you moved to github :)
https://bugs.ecmascript.org/show_bug.cgi?id=2827
I'll happily create a pull request :)