32 sec and 40 sec in Firefox 3 beta 2 on an old Celeron 2.2 GHz with 1.24gb RAM, vs 60-65 sec in Firefox 220.127.116.11 on the same box.
“charCodeAt(i)” != “[i]”
“charAt(i)” == “[i]”
The second implies string alloc where the first returns an integer. I expect it’ll be a wash.
Thanks for pointing that out. However, I do think it will make some difference in IE, because string[i] returns undefined, so they’re not actually executing the intended benchmark and probably something weird and bogus is happening. I will publish 0.9.1 soon with some fixes to the tests.
Since people have asked, here’s what the test categories are about:
access: Array, object property and variable access.
controlflow: Control flow constructs (looping, recursion, conditionals). Right now it mostly covers recursion, as the others are pretty well covered by other tests.
crypto: Real cryptography code, mostly covers bitwise operations and string operations.
math: Various mathematical type computations.
regexp: Regular expressions. Pretty self-explanatory.
string: String processing, including code to generate a giant “tagcloud”, extracting compressed JS code, etc.
The second chart is horrible. I don’t think Tufte would approve. It makes it kind of hard to compare the different browsers
Which is exactly the point of the FIRST graph, directly above it. Sheesh.
I think you missed the point, Jeff. The information shown might be correct, but the style of graph is inappropriate. A regular bar graph (different browsers side by side) would be better than a stacked bar. Nobody really cares what the total effect of running a string process in all of the browsers sequentially would be, because nobody would do it.
Otherwise, interesting post.
I don’t like the way you’ve stacked your graphs. Makes it really hard to compare visually the different individual tests. And does that axis jump from 6 to 16 seconds? nice!
I have use Array.join when concatenating strings. I’ve used more than one platform that exhibits the IE style slowdown when repeatedly concatenating strings with operator+, whereas Array.join has always performed well enough, with fairly linear performance degradation (although it is not the fastest in some cases).
I with you on the syntax front… arr.push wins over str+=otherStr for me
I did a quick test in IE… concatenating a 7 char string 20000 times:
with string concatenation (+=) :1340ms
with Array.push/Array.join :62ms
From your article: “The browser market is more competitive than it has been in years, with Opera 9.5, Firefox 3, Safari 3, and IE 8 all vying for the coveted default browser position.”
I just wonder how come you\re saying IE8 is competing to anything while it’s not even out? Actually, it’s not finished, i haven’t even seen any betas yet. I’d argue that as long as it’s not an actual product in peoples hands, it’s not competing with anything.
I love this blog but you’re saying some “odd” stuff lately. I hope it was just a typo.
Why hate on the font thing? What do fonts have to do with performance? It’s a good article except for that comment.
I think you missed the point, Jeff. The information shown might be correct, but the style of graph is inappropriate.
OK, fair enough. Valid points, all. I changed the chart.
- The top graph and bottom graph are mirrors. So if you don’t like the bottom graph, look at the top one.
They are both terrible graphs, but at least the top one gives you a visual indication of the overall performance of each browser in the total set of tests.
The bottom graph may as well be a table of numbers with '70s-ish colors splattered on it. The visual cues are at best misleading, and generally don’t help a bit. After studying the numbers a while you can figure out what it’s trying to tell you, but it misleads you in almost every way possible (for instance, the numbers on the bottom of the graph don’t help because the browsers are stacked one after another with no relation to time). My initial impression of the first bar on the graph was that opera was the slowest and firefox the fastest on the string test, in opposition to the first graph. Only be reading the numbers and taking in the rest of the graph did I realize that the position of the browsers was the same across bars and had nothing to do with the result of the test in question.
It would have been better if the bar’s full length was that of the longest time and the browsers ordered from fastest to slowest, essentially merging the more standard display of 4 bars in a group (one next to the other) into 1 multi-colored bar that still had a relationship to the scale at the bottom of the graph, rather than stacking them end to end in the same order on each line.
bah, figures you’d change it while I was typing (I was slow). Too bad I don’t really find much of interest in the actual benchmarks, though, other than the usual cautions that come along for the ride when someone releases a benchmark for their own products. At least, for once, Apple actually gives you the tools to run the benchmarks for yourself. They used to be pretty terrible about posting benchmark results with little or no information to make them repeatable.
I just did the test in Konqueror. Too bad I can’t actually compare them to anything, since my Firefox just complains that some script is taking too long to execute and then it wants to kill them. Is that how it’s supposed to work?
Maybe it’s because I’ve been reading Edward Tufte lately, but I don’t think the second graph makes much sense at all. Browsers are stacked together, yet add up to a total time? Just in case I wanted to see what would happen if I gave the same task to all browsers and see their total? And the scale jumps from 6s to 16s, corrupting the data view - if a comparative graphic is not going to use a consistent scale, the data would be better presented as a table.
At minimum I would stack the bars for each browser. As it data displays now, I don’t know what I’m supposed to get out of it.
You’ve misspelled “Xaprb” in the test name.
Are you talking about the GWT (Google Web Toolkit)?
I just ran it on a few machines here and was intrigued by the results. When I get more time I’ll do a write-up on my blog, but until then…
Out of a Sun v250 (UltraSPARC w/Solaris), HP server (P-III w/ Win2k3), and an Apple Powerbook (PowerPC), and a whitebox x64 running Solaris (all very different boxes indeed!):
- Safari on the Mac
- Firefox on x64 Solaris
- Mozilla on SPARC Solaris
- Mozilla on x64 Solaris
- Firefox on SPARC Solaris (threw errors during the test)
- Firefox on Windows (threw errors during the test)
- IE on Windows