Mozilla apologizes for recent add-on disabling issue and provides details
The last week has not been great for Mozilla. Last Friday, reports started to come in from around the world that installed add-ons would not verify anymore and were disabled as a consequence. Users could not download and install add-ons from Mozilla AMO anymore either.
Latest figures show that about 60% of Firefox users install add-ons in the browser; any issue affecting 60% of the user base, especially when it comes to personal choices made by those users, is as critical as it gets.
Mozilla fixed the issue quickly for most users. Quickly still meant that some users had to wait days for their add-ons to work again while others, especially those on older unsupported releases, will still have to wait a bit longer before patches are provided.
Mozilla apologized on the official blog of the organization today. The company acknowledges that it failed and that it is sorry about what has happened. The organization used the Shield service to deliver fixes to users quickly. Since it required the enabling of Telemetry in the browser, it meant that data would be recorded.
Mozilla announced in the post that all Telemetry and Studies data collected between May 5th and May 11th will be deleted.
In order to respect our users’ potential intentions as much as possible, based on our current set up, we will be deleting all of our source Telemetry and Studies data for our entire user population collected between 2019-05-04T11:00:00Z and 2019-05-11T11:00:00Z.
Mozilla CTO Eric Rescorla published a technical analysis of the issue on the Mozilla Hacks blog. He provides insights into Firefox's add-on signing functionality.
The root certificate is used to sign a new intermediate certificate, and the intermediate certificate is used to sign end-entity certificates which in turn sign individual add-ons.
The intermediate certificate needs to be renewed every few years, and it is this renewing that did not happen.
Each certificate has a fixed period during which it is valid. Before or after this window, the certificate won’t be accepted, and an add-on signed with that certificate can’t be loaded into Firefox. Unfortunately, the intermediate certificate we were using expired just after 1AM UTC on May 4, and immediately every add-on that was signed with that certificate become unverifiable and could not be loaded into Firefox.
Mozilla decided to generate a new certificate and install it remotely in Firefox to address the issue.
A post mortem is in the works and will be released soon. Mozilla promises that it will include a list of changes the organization plans to make to avoid any critical issue like this in the future.
Rescorla thinks that this should include monitoring of any "time bomb" components in the Firefox web browser to address any issue before it reaches users, a new system to push updates to users that does not require Telemetry/Studies, and also a look at the add-on architecture.
I think we can all agree that something like the add-on disabling issue should never have happened in first place. It did happen, unfortunately. Mozilla reacted quickly to address the issue. Yes, some users would have liked a better information policy of faster updates, but Mozilla did not really have that many options to fix the issue quickly, especially since it happened over a weekend.
Mozilla needs to implement safeguards to make sure that this never happens again. The organization won't deactivate the entire add-on signing infrastructure of the Firefox browser, that is clear.Advertisement