Created attachment 2
faster sth.js file
I've spent some time reading and trying to improve sth.js.
The main improvement I have made is changing how tests are inserted in the iframe. Currently, an iframe is created, its document is opened and HTML <script> elements are inserted on-the-fly (I will refer to it as the "Doc Stream" method). I have taken the following approach: create a document fragment, create script elements (with document.createElement). When all scripts are here, append the document fragment to the iframe document (later references as "Doc Fragment").
Here are my results (local copy of the website run on Firefox 5):
Doc Stream:
Total Tests Ran: 10865 | Pass: 10587 | Fail: 278 | Failed To Load: 0
Time: 1130551ms (~18m50s)
Doc fragment:
Total Tests Ran: 10865 | Pass: 10585 | Fail: 280 | Failed To Load: 2
Time: 669876ms (~11m06s) (~41% improvement, YAY!)
(measurements have been done only once in an always active tab)
I haven't checked every single test id, but it seems that the failures (and consequently passes) in both approach are the same on FF5.
The only difference is the "Failed To Load: 2". The 2 tests are S15.7.3.2_A1 and S15.9.3.2_A1_T1. After some research, I have noticed that these two tests are the only one triggering the synchronous AJAX load to retrive includes (sth.js ~l100-120). Tests using the same includes, but getting them from the cache ("scriptCache" variable) are not affected. My personal guess is that it's a Firefox-specific bug, because it cannot really be a coincidence.
Other minor changes:
- Now, sth.js requires "textContent" which takes IE8 out of the game. If we really care about making the test suite run on IE8, maybe that "innerText" can be used as fallback.
- the "iframe" variable has been moved to this.run making "testFinished" independent of how tests are run. The iframe is now removed from the tree in the this.run function
- The "testFinished" function was previously called from the iframe via an additional script. I took rid of it and call the function from the main window (because it doesn't make any difference).
- Previously: var win = window.frames[window.frames.length - 1];
Now: var win = iframe.contentWindow;
Exact same object, but now retrieved in a way that is independent of where the iframe is on the DOM tree (because it doesn't matter).
Please, can you test on other browsers and tell me if it works and if there is a performance win.
After test, Opera 11.11 and Chrome 12 aren't affected by an inability to load the 2 tests, confirming the idea that it would be Firefox-specific.
To follow progress on Firefox bug fixing: https://bugzilla.mozilla.org/show_bug.cgi?id=667227
Created attachment 3
faster sth.js file
About https://bugzilla.mozilla.org/show_bug.cgi?id=667227 , it turns out it's an issue with about:blank iframes and synchronous xhr. It's being worked on at https://bugzilla.mozilla.org/show_bug.cgi?id=543435
Meanwhile, the new attachment has a workaround (doing the include loadng before creating the iframe which doesn't change anything functionnally speaking).
I have done more performance measures:
-----
On FFAurora (6)
Docstream: 254504
Docfrag: 215367
On Chrome 12:
Docstream: (1) 445254 - (2) 389022
Docfrag: (1) 390611 - (2) 393119
Opera 11.11
Docstream: 1185647
Docfrag: 720235
-----
Chrome is not very convincing. The second run even showed the docstream method being slightly faster.
I have no way to prove it, but I'm under the impression that the garbage collector has an important influence on the actual performance.
This is not very formal measurements, but seems to give a tendency anyway of the docfrag method being faster.
This seems like a quite useful idea, so I implemented this change, and tested it in modern browsers (Chrome 36, FF 31, IE 11).
I do not see a meaningful speed improvement; either the script loaders have been improved or the current version of sth.js negates the effect of this improvement.
Recommend Resolve - Wontfix
Actually, the picture is more complicated than my previous post suggests.
Running more thorough tests, I find
- a large speedup (0.6 mins -> 0.1 mins) with FF
- a moderate speedup (0.3 mins -> 0.2 mins) with IE11
- a slight slowdown (0.2 mins -> 0.3 mins) with Chrome
I have been using ch08, ch09, ch10 (about 700 tests) as my benchmark.