Mozilla, Google and Apple are building a new browser benchmark
Mozilla, Google and Apple have announced that they are working together to build a new browser benchmark. The service will be called Speedometer 3.
You may have heard of Speedometer, which is hosted on Browserbench.org, it runs a series of simulated tests to measure the responsiveness of web apps. It's worth noting that these are computational tasks, and results often differ based on the platform (operating system and browser) that you use. The upcoming version of the benchmark is designed to measure real-world browser performance on the web.
Mozilla, Google and Apple are building a new browser benchmark
The Speedometer 3.0 project has 3 participants: Blink/V8, Gecko/SpiderMonkey, and WebKit/JavaScriptCore, i.e. the underlying cores of the browser made by Google, Mozilla, and Apple. Interestingly, Microsoft is not a part of this cross-industry collaborative effort, but given that its Edge browser relies on Chromium's Blink and V8 engines, perhaps the company's absence at the table won't matter much.
Engadget points out that the Speedometer's Joint Governance Policy will require a nod of approval from at least two of the participating projects, for any non-trivial change made to the benchmark's code. The current iteration of the tool was developed by Apple's WebKit team. The problem with this is, it wasn't designed to be a universal tool. As noted above, the major browsers in the market have different rendering engines, and script engines. So, it is quite difficult to benchmark their performance accurately.
We have seen browser makers occasionally run a benchmark and claim that their program is superior to the others. But, what is this based on? That's the real problem, these numbers don't truly reflect the actual performance, and merely act as a marketing ploy to convince people their browser is ahead of the rest.
That's why the next version of the benchmark, Speedometer 3.0, is being developed co-developed by the three major players, to ensure a certain level of fairness, or a standard if you will. The browser makers will share the code for the benchmark, and non-trivial changes made to it will include anything that impacts the result such as test cases, test runners, bug fixes, and the appearance of the tool.
Trivial changes such as modifications to the documentation on the other hand will not require the approval of the other parties. All major changes that are made to the project will require a consensus, i.e., everyone must agree to the decision.
The internet is an ever-evolving space, older technologies are replaced by new standards, which are faster, consume less resources, etc. Benchmarks need to catch up with those, relying on an old version (based on old standards) will deliver inconsistent, and inaccurate results.
This collab project will allow developers to understand how to optimize the performance of their browser, for example, the upcoming benchmark will represent real-world workloads such as various JavaScript frameworks. This in turn could help improve the cross-browser and cross-platform performance, and in a sense, have a greater impact on the internet itself.
References: Apple, Mozilla and Google.
The GitHub page for the project states that Speedometer 3 is unstable as it is still in early development, and recommends users to use Speedometer 2.1 for now. It's not clear what the benchmark is capable of, but seeing the browser makers come together is not a common sight, I view this as a positive sign.
Instead of investing such a considerable development effort as it happens sometimes to earn tenths of seconds, maybe those browser developers could simply consider shipping by default with uBlock Origin (for those who haven’t banned it yet) for a zero-effort considerable increase in performance that would double with fighting monstrous privacy nuisances ? Surprisingly, Apple used the performance excuse to… kick out uBlock Origin, while Google is about to do the same thing.
Lose lose situation for Mozilla anyways, Google and Apple will ignore any changes from them because they’re big boys, in the end it will become a big biased benchmark for Chrome-based browsers.
The problem with a 3 way system is that it’s not going to lead to any real discourse. What can and may happen is that some 2 strong opinions will override the one regardless of if they have a sound reason to disagree. This is why things are the way they are. Now if there were 4 entities you may have a hope of some real discussions. At the end of the day the entire internet and landscape for browsing the internet and how websites are developed is dictated by Google anyway so events like this to pretend there is some sort of middle ground is absurd.
From an article in couple of months ago:
“Mozilla criticizes Google, Apple and Microsoft for using their operating systems to force users away from other browsers.”
Shall I then presume, they have kissed and made up with one another? Aw, just in time for the Holidays. [Tears]
Both Google and Mozilla Firefox as browsers and M$ as OS seriously believe that they are in fact THE internet Marshal,,,,at the very least they track you to see if you’re a bad guy, with an assortment of allegedly nefarious behaviors. ( I’m an old guy, what can i do? )
I can be on the internet for 10 minutes Chromium or Gecko, run bleach-bit as root, purge a half a gig of data-waste?, then turn right around and run bleach-bit as non-root and delete about an additional 1.2 Gigs of data.. Not to mention they love your very fast internet bandwidth, oh…and all that ram you just upgraded….they’ll will use it believe me….when my internet begin to buffer and lag….guess who is using it thus burdening my endeavors…YUP it’s them. ALL OF THEM.
Users should offer a complimentary, ” your welcome ” to the Internet Marshal’s for they have much more access to your system resources then you do.
How the World as such is dependent on a select few US corporations.
@Benjamin
Mozilla, almost completely funded by Google, is in permanent rebellion against Big Tech. Or so I was told here on gHacks.
https://www.ghacks.net/2021/06/07/webextensions-community-group-formed-by-apple-google-microsoft-and-mozilla/
What are you talking about? What does the world being dependent on a select few US corporations have anything to do with the Mozilla Foundation. Stop letting Firefox live rent free in your brain. The world does not revolve around you and Firefox,
All browsers are pretty similar in speed. I don’t really spend much time benchmarking anything.
I always noticed Firefox being a tad slower than any of the Chromium browsers. But not by a significant margin. Probably good that Speedometer 3.0 is being collaborated with the three. That way you get a reasonably accurate benchmark. If you really want to compare speed.
If privacy and security are my main concerns speed is far less and when I see the speed test score performed with Speedometer 2.1 (42.0 +- 0.70 on a board displaying a 140 upper limit) I don’t feel bad about it : no Ferrari here, obviously. Gosh : what would it take to reach 140? A super calculator?!
Nevertheless if we get three companies agreeing (with a 2/3 majority requirement to modify the protocol algorithm) for an updated and enhanced relevant Speedometer 3.0 then why not? Even if it is really not my cup of coffee.
These benchmark scores are not an indicator of browser engine superiority but rather of user device. More expensive device with better processor will score more regardless of browser. In other words waste of time, unless a particular user wants to test own device to check if it is working properly compared to its peers.
i’ve found the benchmark useful for comparing the major browsers (firefox chrome brave chromium vivaldi opera edge) on my machine. in my experience firefox scored 100 pts lower than the competition i analyzed.