Mozilla apologizes for recent add-on disabling issue and provides details
The last week has not been great for Mozilla. Last Friday, reports started to come in from around the world that installed add-ons would not verify anymore and were disabled as a consequence. Users could not download and install add-ons from Mozilla AMO anymore either.
Latest figures show that about 60% of Firefox users install add-ons in the browser; any issue affecting 60% of the user base, especially when it comes to personal choices made by those users, is as critical as it gets.
Mozilla fixed the issue quickly for most users. Quickly still meant that some users had to wait days for their add-ons to work again while others, especially those on older unsupported releases, will still have to wait a bit longer before patches are provided.
Mozilla apologized on the official blog of the organization today. The company acknowledges that it failed and that it is sorry about what has happened. The organization used the Shield service to deliver fixes to users quickly. Since it required the enabling of Telemetry in the browser, it meant that data would be recorded.
Mozilla announced in the post that all Telemetry and Studies data collected between May 5th and May 11th will be deleted.
In order to respect our users’ potential intentions as much as possible, based on our current set up, we will be deleting all of our source Telemetry and Studies data for our entire user population collected between 2019-05-04T11:00:00Z and 2019-05-11T11:00:00Z.
Mozilla CTO Eric Rescorla published a technical analysis of the issue on the Mozilla Hacks blog. He provides insights into Firefox's add-on signing functionality.
The root certificate is used to sign a new intermediate certificate, and the intermediate certificate is used to sign end-entity certificates which in turn sign individual add-ons.
The intermediate certificate needs to be renewed every few years, and it is this renewing that did not happen.
Each certificate has a fixed period during which it is valid. Before or after this window, the certificate won’t be accepted, and an add-on signed with that certificate can’t be loaded into Firefox. Unfortunately, the intermediate certificate we were using expired just after 1AM UTC on May 4, and immediately every add-on that was signed with that certificate become unverifiable and could not be loaded into Firefox.
Mozilla decided to generate a new certificate and install it remotely in Firefox to address the issue.
A post mortem is in the works and will be released soon. Mozilla promises that it will include a list of changes the organization plans to make to avoid any critical issue like this in the future.
Rescorla thinks that this should include monitoring of any "time bomb" components in the Firefox web browser to address any issue before it reaches users, a new system to push updates to users that does not require Telemetry/Studies, and also a look at the add-on architecture.
Closing Words
I think we can all agree that something like the add-on disabling issue should never have happened in first place. It did happen, unfortunately. Mozilla reacted quickly to address the issue. Yes, some users would have liked a better information policy of faster updates, but Mozilla did not really have that many options to fix the issue quickly, especially since it happened over a weekend.
Mozilla needs to implement safeguards to make sure that this never happens again. The organization won't deactivate the entire add-on signing infrastructure of the Firefox browser, that is clear.
I changed to vivaldi, I didn’t like that a fundamental feature broke websites and the security of my web browsing.
I am not looking back, no technical babble or bulldust just a plain old user for over a decade who got sick of a browser that is just a chrome replicator with little to no new features that are original.
why do the tech companies not generate new certs before they expire?
@ dwa
Seems, Mozilla Corp had weaponized Firefox add-on/extension signing against users who stayed back with Legacy add-ons in FF 56 and FF 52 ESR, in order to push them onto FF 66.0.4 or FF 60.6.3 ESR.
……. Also, it seems FF 66.0.4 has only forced automatic update, ie no more option for manual update = FF Update has been weaponized by Mozilla Corp, eg to later impose forced Telemetry and/or Studies.
This is similar to how M$ has weaponized Windows Update against Win 7/8.1 users, in order to push them onto Win 10, eg processor-blocking updates in the monthly Patch Rollups, Patch Rollups that degrade the Win 7/8.1 experience, etc.
well. that’s all quite irrlevant. the point is, if the cert hasn’t expired none of it would happen. it doesn’t just apply to moz. we’ve seen it before elsewhere. so why is there no solution to the whole expiring thing?
>> especially those on older unsupported releases, will still have to wait a bit longer before patches are provided..
An absolute lie. Typical Silicon Valley con job speak of the “new” Generation Z that hide information when the fix can be done in 1 hour. There is a fix on-line, it was posted on this forum, many installed it, it was simple concise and even required a bit of coding for the novice—-for “unsupported releases”, and it works.
What this was was a in-cahoots con job with Mozilla to upgrade to FF 66, 67, 68 and dump the rest, even the makers of Waterfox (all of a sudden releasing WF 58 based on FF 57, when everyone uses FF 56). ghacks dot net were also in on this, pretending to act like innocent bystanders—I have no doubts given the Fake News and deletion of accurate comments simply because the moderator doesn’t like it. When there’s a fix, report it.
Oh no —- “Mozilla fixed the issue quickly for most users”. Not true. You mean, through about:studies, Mozilla fixed the issue quickly by monitoring users.
I’ve been a Mozilla user for nearly 20 years, I’m currently testing out Iridium and fear it may be the end of my time as a firefox user.
@Anonymous:
I ran Iridium as a backup browser for a while, but I don’t think it’s very actively developed, given how infrequent updates were. You *might* be better off with a different Chrome/Chromium fork. But I could be wrong, and maybe things have changed since I uninstalled it.
Mozila created a new certificate, but shipped one that was created back in 2015 (Friday, April 3, 2015) to resolve the issue?
If Mozilla can push updates to users, why can’t it push a NOTICE so that people don’t freak when something like this happens?
I woudn’t have minded as much if I knew from the outset that there was an issue with addons, but the way it happened made it seem like there was something wrong with MY instance of Mozilla, and spent hours trying to find a fix.
At least Mozilla had the decency to admit fault which is quite unusual in this time and age. Accept it, forget about it and move on. By the way, I was hit too with the addon chaos.
Can’t get more obvious then a expiration date? If you can’t keep track of when they expire that’s being incompetent and begs why this would happen? I could excuse a outright failure that could not be predicted. As people say shit happens, but knowing the impending doom date and not being prepared has no excuse.
@John IL:
It’s called a “tickler,” Mozilla, and lawyers have been using them to avoid missing deadlines since before computers were invented. Look into it. Maybe just set yourself a recurring reminder on Google Calendar or something…
Same thing happened to Tor browser, I rarely use it but yesterday fired it up and after a little while it disabled extensions, luckily Tor had just released an update on 6th May for regular Tor and 8th May for alpha.
https://blog.torproject.org/noscript-temporarily-disabled-tor-browser
This sort of thing is endemic to the software industry – programmers and program manager more concerned with adding features to “beat the competition” and stroke their egos as developers than be concerned about security, reliability and usability.
This won’t change until something happens that ends up killing a few thousand people (but apparently even the Boeing crashes haven’t resulted in any change.)
My Firefox 66.0.3 on openSUSE LEAP 15 (and all previous Firefox releases) has the incredibly annoying habit of deleting the Web page file name when you change the directory during a “Save Page As”. I can’t tell if this is a Mozilla issue or a branded openSUSE issue in terms of the file same dialog. But that doesn’t matter. The fact that it happens at all and has been happening for years proves major incompetence on the part of the maintainers. And since I save tons of Web pages, it means I have to constantly do a copy of the Web page title before saving the page. I frequently forget, then have to cancel the save, copy the title, then do the Save Page As a second time. I’ve filed a bug report with Mozilla, but they can’t reproduce it. I guess I’ll have to complain to openSUSE and see if they can fix it. But the fact that it happens at all is a condemnation of everyone involved on both sides.
For me, all of this “news” doesn’t make sense because a simple upgrade to a beta version of Firefox eliminated the problem. All of my add-ons worked perfectly, albeit, there was a yellow warning that the add-ons were “experimental” [or some such language] and that I should be careful in their usage because Mozilla hadn’t “verified” them [or some such language].
The only users affected, then, were those who were using a current build.
What does it all mean? That the only the current build of Firefox can be affected by an expired certificate and all other builds are exempt from such security measures? Portable and earlier?
I wasn’t using a current version of FF, on linux and I WAS affected.
Oops….so that’s what happened..?? I opened FF one day and all, and I mean all my add-ons were disabled .I ended up looking for and adding similar ones and I was basically up and running again..The very next day I find all my old add-ons had come back ..!! I’ve had to disabled the “new” ones and gone back to my “old” ones…. :)
Can someone explain why addons need certificate / signing ? It could have avoided this kind of fiasco in the first place.
Corporate control.
All I know is that I upgraded from 66.0.3 to 66.0.5, replaced my profile with a backup from a month ago just in case, and now none of my extensions will update. I know that at least two have newer versions, but when I manually check for updates, none are found.
This is actually a plot to gather telemetry data during the fiasco because Mozilla want to know the info of power users who have disabled telemetry.
Aditionally, now Mozilla has excuse to introduce one more backdoor to ‘update’ Firefox. I wonder why they can’t just install hotfix update using the update function like other softwares do?
Do you wear a tin foil hat?
how nice of them to apologize for such a gargantuan blunder
all is forgiven!
for a blunder that affected the vast majority of users with Mozilla extensions, they should have fix it faster
it took about 3 days for an official fix
As with FF 66 even after several tricks I can’t stop auto-update (still a f…..g error popup @&#$!) I finally switched to chrlauncher-win64-stable-ungoogled.
So they still may not give users back control to keep addons that have a expired certificate. Certificates should in most cases never be allowed to expire and only be allowed to be revoked when abuse is detected.
If you are in a situation with no internet connection (remote area, disasters, ISP problem) and evrey 24h Firefox just checks the date to see if the certifcate is expired and then disables addons if the certicate is expired, then your addons will fail. This is not a problem for addons that are needed for internet websites and browsing (as you have no internet access), but it is for addons that have offline productivity functions or are used for local websites on your intranet.
The funny thing is that this problem exposed a lot of people (who were whining about Firefox’s new changes and saying they’re done with Firefox and moving to other browsers) are still using Firefox to this date!
Semantically the two parts (preceding and following the brackets) don’t fit together.
As for me, it wasn’t a big jump from Fire to Water (FF56 to Waterfox) – how many others have permanently switched, IDK.
I hope Waterfox ASO get a boost from this, and don’t follow the design and functionality choices of post-56 Firefox. No extreme “Metro Style” fot me, please!
When I read FF user comments like here below, that with “FF 66 even after several tricks I can’t stop auto-update”, then it’ll surely be permanently – I’m here in Africa with no flatrate etc. and don’t tolerate the browser forcibly sucking up my credit/bundle & bandwidth.
@WAB-Flex
Mind you, it’s a different version of Firefox people are using. People gripe about new spyware-loaded versions, and that they have to find remedies here themselves.
the mozilla firefox team turn the best browser to crap first time see new version create more bugs!! dont have beta testers? mozilla collect private data from users thats the reason create this bug that is true.
Except they didn’t say sorry at all. The closest was “we let you down” that’s not the same as saying sorry.
“We strive to make Firefox a great experience. Last weekend we failed, and we’re sorry.”
Martin provided a link in the fourth paragraph. ;)
I’m upset about what happen just as much as anyone else, but quite frankly I think that’s enough with the ranting, particularly when is bringing nothing to the conversation and not even remotely close of being true!
“We strive to make Firefox a great experience. Last weekend we failed, and we’re sorry.”
Literally from the first paragraph, did you even bother to check the blog page at all?
5 days ago on the “What Mozilla needs to do now (after cert add-on disabling disaster)” post on Ghacks I wrote in the comment section –
”
If Mozilla wants to still be around 5 years from now they should:
#1 – Get on Twitter and grovel. Acknowledge how big a F-up this is. Tell us how you are going to bake in a solution to override any stupidity you force on the user on purpose or accidentally so we never have to go through this kind of nonsense ever again. ‘I’ installed a browser on ‘MY’ computer – ‘YOU’ (Mozilla) should not have the ability to disable it for any reason ever – not even by accident.
#2 – Understand and listen to the users. For 10 years now they have ignored their loyal user base rather than build on them. We don’t want baked in Pocket – we want baked in UBlockOrigin. We don’t want data collecting schemes, telemetry, and forced advertising experiments we have to x100 disable in the about:config – we want clean out of the box performance (market yourself as the privacy browser with user controlled ads and you will gain +20% market share almost overnight). We don’t want the hamstrung addons we have now – we want robust addons (like before) so Build out the extension API system already. It’s been over a year now and addons are still basic.
#3 – Get on social media and show the users you are working on the things they want. Tell us you just finished on an API that will let them do XYZ. Tell us you are in the middle of making pages load 10% faster. Tell us you’re taking out screenshot but your are putting back rss preview because you realized it was a stupid mistake. Tell us you realize recognize your mistakes and how you are fixing it.
#4 – Fire all their corporate management idiots and hand control of the development back to the coders. Every marketing decision I’ve seen from them resulted in quick cash but loss in user base. No users = dead browser / company. Fast “creative†revenue is sinking the company long term.
”
I am slightly encouraged by the news of Mozilla’s response. They’re acknowledging the F-up. They’re explaining the F-up. They are working on a solution for no more (of this kind of) F-ups. They’re paying new attention to the addon architecture (a main draw for FF users). Most importantly they are proactively respecting the privacy of the users by deleting the telemetry we were forced into giving up. They’ve done all the short term stuff I suggested within a few days. I am cautiously optimistic and feel a little better about the crap we just went through.
@Hm Mozilla: “I am slightly encouraged by the news of Mozilla’s response. They’re acknowledging the F-up. They’re explaining the F-up. They are working on a solution for no more (of this kind of) F-ups.”
I would be more hopeful if this was the first this happened. But it’s not. It’s the second, and they didn’t seem to fix the underlying process issue after that time. This one just affected a lot more users.
I thought a renewed certificate will automatically solve the add-on/extension signing problem caused by the expired certificate, ie as soon as Firefox next auto-connects to Mozilla’s servers to check on the renewed add-on/extension certificate which happens once every 24 hours = affected FF users will just have to wait for the problem to auto-clear by itself. Why the need for an update to the browser or the installation of an extension to fix the add-on/extension signing certificate problem.?
Maybe, there is a preference setting in about:config to force FF to immediately connect to Mozilla’s servers to check for the renewed add-on signing certificate.
.
AFAIK, when an expired SSL/TSL certificate borks an https website to become inaccessible by the major browsers, a renewed certificate will automatically make the website accessible again, ie no need any update to the browsers.
Something fishy is going on.?
@AnorKnee Merce:
A lot of Firefox users don’t allow the browser to automatically update, so that’s not a complete enough solution.
5 days ago on the “What Mozilla needs to do now (after cert add-on disabling disaster)” post on Ghacks I wrote in the comment section –
”
If Mozilla wants to still be around 5 years from now they should:
#1 – Get on Twitter and grovel. Acknowledge how big a F-up this is. Tell us how you are going to bake in a solution to override any stupidity you force on the user on purpose or accidentally so we never have to go through this kind of nonsense ever again. ‘I’ installed a browser on ‘MY’ computer – ‘YOU’ (Mozilla) should not have the ability to disable it for any reason ever – not even by accident.
#2 – Understand and listen to the users. For 10 years now they have ignored their loyal user base rather than build on them. We don’t want baked in Pocket – we want baked in UBlockOrigin. We don’t want data collecting schemes, telemetry, and forced advertising experiments we have to x100 disable in the about:config – we want clean out of the box performance (market yourself as the privacy browser with user controlled ads and you will gain +20% market share almost overnight). We don’t want the hamstrung addons we have now – we want robust addons (like before) so Build out the extension API system already. It’s been over a year now and addons are still basic.
#3 – Get on social media and show the users you are working on the things they want. Tell us you just finished on an API that will let them do XYZ. Tell us you are in the middle of making pages load 10% faster. Tell us you’re taking out screenshot but your are putting back rss preview because you realized it was a stupid mistake. Tell us you realize recognize your mistakes and how you are fixing it.
#4 – Fire all their corporate management idiots and hand control of the development back to the coders. Every marketing decision I’ve seen from them resulted in quick cash but loss in user base. No users = dead browser / company. Fast “creative†revenue is sinking the company long term.
”
I am slightly encouraged by the news of Mozilla’s response. They’re acknowledging the F-up. They’re explaining the F-up. They are working on a solution for no more (of this kind of) F-ups. They’re paying new attention to the addon architecture (a main draw for FF users). Most importantly they are proactively respecting the privacy of the users by deleting the telemetry we were forced into giving up. They’ve done all the short term stuff I suggested within 1 week. I am cautiously optimistic and feel a little better about the crap we just went thruugh.
> They’ve done all the short term stuff I suggested within 1 week. I am cautiously optimistic
There is no reason to be optimistic at all, don’t fall for their PR manipulation. This is not a change in their corporate culture, they have addressed almost nothing of this former comment and they have often even confirmed that they won’t ever : the right to override signature enforcement, removing Pocket, including uBlock origin, making telemetry and the studies backdoor easy to disable fully or better opt-in, stopping advertisement in browser, making extensions powerful again, rss support, stopping the management stupid marketing decisions…
Worse, they are going to include a new silent backdoor besides Studies, with the claimed purpose of pushing quick fixes, that should instead be pushed with visible releases in my opinion. They have abused the Studies delivery channel before to push adware like Mr Robot while it was supposed to help development only. I won’t trust them for the new channel either and will disable it (if they allow me to do it, which is not obvious).
Hear hear. All of this reeks of what’s yet to come.
Much ado about nothing. Firefox is just a browser that can be used fine without add-ins (like 40% do). A browser isn’t a productivity too. It is not like Microsoft’s BSOD with every monthly updates that crashes PCs and inflicts Trillion of $$ in damages.
There are 100 other browsers to use until Firefox is fixed.
@ilev: ” A browser isn’t a productivity too.”
It is for a lot of people. But regardless, what does it matter whether or not browsers are classified as a productivity tools?
“A browser isn’t a productivity tool.”
Speak for yourself. I regularly use my browser for research, and development and testing if I am working on a project that involves web (or browser) development. Depending on what I’m doing, add-ons can be essential to my workflow.
> apologizes … provides details
That’s PR-speak for “finding excuses”. Its deja vu from Facebook’s playbook: same we-are-sorry-we’re-caught lather-rinse-repeat.
> The organization won’t deactivate the entire add-on signing infrastructure of the Firefox browser, that is clear.
Human translation: “what can we disable next time eh?”
Switched to Brave from using Firefox since begining of time. All addons are Chromium compatible and the speed of Brave is out of this world. At least for me.
@Be polite:
I was very enthusiastic about Brave until I read this:
Brave Browser Sacrifices Security | Netsparker
https://www.netsparker.com/blog/web-security/brave-browser-sacrifices-security/
* * *
[A]round three years ago, code was added to the browser’s Github repo to whitelist certain third party domains. The Facebook URLs that are used to track users were included in this whitelist.
* * * *
Facebook is one of the most notorious nonconsensual user-profile builders on the Web. From what I’ve read, even if you’ve never had an account, there’s a very high chance they have a profile on *you*, from running third-party-domain scripts on non-Facebook sites you visit.
In browsers that support it, I’ve blacklisted Facebook in NoScript and will continue to do so when I’ve switched to uMatrix/eMatrix. My understanding is that users *cannot* blacklist whitelisted domains in Brave. Facebook is apparently blocked from setting or accessing cookies in Brave, but I’m not as sure as Brave’s spokespeople that Facebook can’t track users through fingerprinting. Unless I see a convincing argument that I’m wrong to be concerned, this is a deal-breaker for me.
As always, I’m happy to be corrected.
@Be polite: yada yada…. ;)
“the speed of Brave is out of this world”
After installing and setting up Brave as close as I could to my other chromium browsers, minus the ad blocker, I then looked at the page load times of a couple websites using the Dev Tools. All of the browsers had javascript and cache enabled, uBO using the same filters and rules.
CNN:
Firefox: 2.31s – 88 network requests.
Chrome: 6.85s – 144 network requests.
Brave: 5.75s – 127 network requests.
Initially both Chrome and Brave looked like they were going to have very fast times but every time I thought they were done more network requests were made.
PCWorld:
Firefox: 1.61s – 96 network requests.
Chrome: .584s – 115 network requests. 3.46s after it decided to load about:blank. Vivaldi did the same thing.
Brave: .344s – 111 network requests. Now that was impressive!
I’ve uninstalled Brave, I’ll just keep Vivaldi as my default chromium browser. Also, Brave has more dropped frames watching 1080p 60fps video on YouTube than Chrome does but less dropped frames than Vivaldi which dropped 12 in one minute, not bad. FF is always at zero dropped frames on my Win7 Pro x64, 4 core, discreet graphics, 16GB ram desktop.
It’s pretty darn hard to make a bad choice with desktop browsers, I think Chrome is the worst choice for privacy but performance wise they are all pretty good. On my system FF has better font rendering, much better smooth scrolling, excellent graphics performance and video playback. Page load times are a toss up.
Me too, changed to the Brave…
I’ve used Brave, and it’s for now my second browser choice until Chromium Edge is fully released. I have not noticed any difference in speed, what I did notice is that Firefox actually uses less RAM and displays fonts much better than Brave or any other Chromium fork. Not to mention that Firefox add-ons are much better than Chromium ones, due to AMO’s privacy policies. Also uBO is much better on Firefox too.