Firefox does have a faster rendering engine though, which in turn balances some of these slower results.
I figured the geometric means as mentioned above and Opera is still the fastest (Opera performance = 1.0)
Safari = 1.56 (56% solwer), Firefox is 1.96 (96% slower) and IE7 is 2.82 (182% slower).
Same conclusion as the original bar charts, however, Safari doesn’t do as well, and IE7 isn’t quite as bad. Also IE7’s worst weaknesses are the base64 and validate-input tests that are 72 and 51 times worse than Opera respectively.
Also, comparing beta versions vs released products is the oldest benchmarking trick in the book. Compare released version with released version for a true comparison.
@Jeff: as always there may be too many factors playing in here to make this an “absolute” result. However, as a JS system lib coder it provided some useful hints, and your results matched my experience regarding the browsers you tested. Thank you for spending some time on this.
Frankie
The first chart summarizes the results as a linear sum of the times - length of the bar for each browser. This weights the results more heavily for the longer running tests, notably the string test which takes about 30% of the total in even the fastest browser.
Back in the 80s the geometric mean of the relative performance on each test was adopted as the standard way of summarizing a group of benchmarks to goive equal weight to them all. So the fastest browser on each test scores 1 on that test, and the rest get a score that is the ratio of that browser’s time to the best time. Then you take the geometric (multiplicative) mean of the scores to arrive at an overall performance ratio that doesn’t weight any single test more than the others.
Of course in the real world web pages may use some operations more heavily and that is an interesting topic also.
Jeff: Can you provide some justification as to why you’ve chosen to test three shipping browsers (IE, Firefox, Safari) against one beta browser (Opera)? If you do not have the time to download and test Firefox 3 b2 and a recent Webkit nightly, then you should be comparing against Opera 9.2 as that is a shipping release like the others. Of course, I would prefer if you could add all three of those into the mix (and IE6 if it will even run the test). It would certainly give a more complete summary of where things stand at present.
Very neat approach for the test. Including repeatability and confidence margins was a great idea.
Funnily enough, I had seen another such test (mainly oriented at comparing the new scripting engine in Opera 9.5 to the others) a few days ago :
http://nontroppo.org/timer/kestrel_tests/
Here are the results on a pretty old machine :
Opera 9.22 - 36350.6ms +/- 3.5% (no NaN error)
FF 2.0.0.6 - 78098.2ms +/- 8.0%
Actually the reason the geometric mean is used is that the relative results do not change if another browser is added - the existing browsers maintain their relative rankings. This is not true for the arithmetic mean or adding times linearly in a bar chart. See the paper at:
http://portal.acm.org/citation.cfm?id=5673
Correction - the bar chart rankings would not change obviously if another browser were added, what I meant was that over time things change, and some tests improve and get shorter, so either you have to keep changing the test (and be accused of tuning the test to benefit one browser or the other), or find a normalized means of summarizing results that weights things equally.
On a subjective level, I work on sites that use Extjs extensively, and IE7 is much quicker than Firefox 2.0.0.8 on most of our pages.
The individual tests are balanced to take about the same amount of time, as measured in Firefox 2 and IE 7 (excluding outlier scores). I do not think it makes sense to balance by taking the ratio to the fastest browser on each individual test. That gives a worse penalty for average performance on a test when one competing browser is really fast and another is really slow, than when they all get similar speeds. It also means you could get totally different proportional scores by taking different subsets of browsers. Summation may be simplistic but it behaves in predictable, common-sense ways.
Just did a comparison between Safari 3.0.4 and Webkit Nightly r28949 … amazing, webkit is 2.6 times faster than Safari…!
Safari 3.0.4: 10123.4ms
Webkit r28949: 3912.2ms
Both tests performed on Mac OS X 10.5 ‘Leopard’, iMac, 2.16 GHz Core 2 Duo
I have to agree with these results. We have two workloads where we notice that IE is much slower than FF: 1. Google maps with lots of pointers. 2. dhtmlXgrid with lots of rows. IE is quite a bit slower.
After seeing this we installed FF on all of the machines in the company and set it as the default browser.
I don’t see this as a partisan issue. I just want to use the fastest / most compatible browser.
I like this benchmark as it shows the specific weaknesses of each browser. Time go get crackin guys! Lets get a JIT compiler in there!
That IE7 string result is crazy…
I wouldn’t be surprised if strings are the most common things used in javascript.
Curious…
Since this test supposedly is concentrated more on pure JavaScript performance, how easy would it be to run under something like the Rhino JavaScript interpreter for the JVM? It would seem to me if it’s pure JavaScript - measuring it’s performance in something dedicated to running just JavaScript would be useful to know.
After seeing this we installed FF on all of the machines in the company and set it as the default browser.
I don’t see this as a partisan issue. I just want to use the fastest / most compatible browser.
Firefox 3.0 nightlies is significantly faster now, even beating out Opera 9.5 beta. And the Mozilla devs aren’t done yet with Javascript optimisations.
Firefox 3, JavaScript is also good
Just an update, on our slowest test system (old), running Vista 32bit, 1gb RAM, IE7 was running around 35000.
Installing IE8 and rerunning the test the number is consistently under 10000.
Curious if anyone else is seeing this dramatic of a change in IE8. I know MS was reoptimizing all the Javascript for IE8, but this is pretty impressive, as IE8 is beating the latest build of FireFox 3 on the same system.
Speaking of IE8, is anyone else noticing rendering problems with Jeff’s page?
Here some results on a MacBook Pro 2,16 GHz 4 GB RAM 10.5.
http://blog.source-code.de/archives/5-The-great-FireFox-3-JavaScript-Benchmark.html
Because Safari 3.1 just came out I thought I’d give it a try with SunSpider on my MacBook Pro with 2.16Ghz Intel Core 2 Duo and 3GB RAM. Got a total time of 3621ms which seems pretty nice. Same machine with FF 2.0.0.12 gave a total time of 13451ms. Opera 9.26 failed on the Regex test section. Minus that test it was a total time of 11665ms. I think I’ll be sticking with Safari for this platform.