Mozilla explains the January 2022 Firefox outage that blocked connections
On January 13, 2022, Firefox users from all over the world started to report connection issues. The browser failed to connect to any site and users were reporting hangs and crashes.
Mozilla published a detailed technical explanation of the incident on the company's Mozilla Hacks website on February 2, 2022.
The organization received reports about Firefox hanging during connection attempts on January 13, 2022. At the time, it saw that crash reports were spiking but did not have much information about what was causing the issue.
Mozilla engineers discovered that a network request was causing the hangs for Firefox users. Engineers looked at recent changes or updates, but did not find any that could cause the issue that users experienced.
Mozilla suspected that the issue could have been caused by a recent "invisible" configuration change by one of the cloud providers that it uses for load balancing. The organization uses the infrastructure of several providers for services such as crash reporting, telemetry, updating or certificate management.
Settings were not changed in inspection, but engineers noticed that the Telemetry service was serving HTTP/3 connections, which it had not done before. HTTP/3 was disabled by Mozilla and users could finally use Firefox again to connect to services. The HTTP/3 setting at the cloud provider was configured with the automatic value.
Mozilla investigated the issue in more detail after the most pressing issue had been taken care of. All HTTP/3 connections go through the networking stack Necko, but Rust components use a library called viaduct to call Necko.
Necko checks if a header is present and if it is not, will add it. HTTP/3 relies on the header to determine the request size. Necko checks are case-sensitive. It now happened that the requests that passed through viaduct were put into lower-case automatically by the library; this meant that any request through viaduct that added a content-length header passed Necko but ran into troubles with the HTTP/3 code.
The only Rust component that uses the network stack and adds a content-length header is the Telemetry component of the Firefox web browser. Mozilla notes that this was the reason why disabling Telemetry in Firefox resolved the issue on the user side. Disabling HTTP/3 did also resolve it.
The issue would cause an infinite loop, which blocked all further network communication because "all network requests go through one socket thread" according to Mozilla.
Mozilla states that it has learned several lessons from the issue. It is investigating all load balancers and reviewing their configurations so that future issues like that can be avoided. The deployment of HTTP/3 at Google, which was the cloud provider in question, was unannounced. Lastly, Mozilla plans to run more system tests in the future with "different HTTP versions".
Closing Words
Mozilla reacted quickly to the emergency situation and has resolved it. It may have damaged the reputation, and some users may have switched to a different browser in the process. Mozilla should ask itself whether it is a good idea to rely on cloud infrastructure that is operated by its biggest rival in the browser space. Some Firefox users may also suggest that the organization looks at the browser's handling of requests to make sure that unnecessary ones, e.g. the reporting of Telemetry or crash reporting, will never block connections the user attempts to make in the future.
Now You: what is your take on the incident?
I can no longer get Firefox to work. I have uninstalled and reinstalled several times.
@ Dorian
> Tor is built on top of a terrible browser just for the
> ‘agencies’ to get easy access to people’s information
You are free to use another browser with Tor you know. Sure, “Tor Browser” has FF but even then, so long as you have Tor Browser running, you can configure and use another browser. You can also use Tor without FF, by installing it yourself and not using the Tor Browser Bundle. There is also TAILS Linux.
In the not too distant future, most software will fail to run by design if they can’t reach ad and tracking servers.
Basically, is what Windows 11 is becoming. Better start learning Linux from now.
You’re probably right, I’m afraid.
Some Google guy who works in the dark, literally, will hear about this for years.
viaduct is an internal Moz library, nothing to do with Google. The problem started and ends with problems in the Moz software.
If anyone is seriously interested in minimizing privacy issues in any browser, not bothering to turn off telemetry, despite having the option to do this in the browser, is asking for trouble. If users were to take these steps, by default practice, to reduce privacy invasive issues, then in this case, and in addition, a flaw could have been avoided as well as increasing security.
The bottom line is, try to do your best to modify your settings, to prevent anything and everything from being used against you.
Peter Newton [London UK]
If they can take your data, you can do whatever you want to try and prevent it. It’s a game, users who don’t play already lost. Not just FF, any online service, OS, whatever.
> Mozilla notes that this was the reason why disabling Telemetry in Firefox resolved the issue on the user side. Disabling HTTP/3 did also resolve it.
The fact that Mozilla developers and testers didn’t discover this problem BEFORE the release confirms that Mozilla employees themselves turn off telemetry (otherwise the problem would have been discovered before the release by themselves).
What an irony!
Nah, they use either use Edge or Chrome not Firefox lol
This reminds me of the addon-ocolyps, which actually wasn’t that long ago.
https://www.ghacks.net/2019/05/04/your-firefox-extensions-are-all-disabled-thats-a-bug/
I wish Firefox was more like Blender, a program that doesn’t do random stuff behind my back, and doesn’t randomly break for no reason.
“Mozilla should ask itself whether it is a good idea to rely on cloud infrastructure that is operated by its biggest rival in the browser space.”
Are U kidding?? Currently they get $400M+ a year for only including all-G-search!!
Installing arkenfox.js is mandatory.
and always will be.
Yeah, as long as you READ what each swich does, just inserting user.js can cause issues. But the potential downsides are all well explained. Make a profile or at least prefs.js backup before experimenting and get ready for lots of reading.
Gives more privacy and obliterates all the ad junk FF and other browsers drown in.
LibreFox is a much simpler way for newbies to get to about the same place; arkenfox is more complete.
> Yeah, as long as you READ what each swich does
Stop being a baby – https://github.com/arkenfox/user.js/wiki/3.2-Overrides-%5BCommon%5D – pick and change the things you need/want
> These few items, out of 140+ pref changes, will solve 99% of usability issues and you only need to do it once
arkenfox is like firefox on steroids – https://privacytests.org/ – and that doesn’t even list all the things that dFPI/FPI covers
https://github.com/arthuredelstein/privacytests.org/issues/80#issuecomment-1008422947
@aaa I agree. arkenfox.js is brilliant
Yes, it is true that arkenfox increases privacy by mitigating state and navigational tracking among other methods. Seems as if Iron Heart is being very disingenuous making claims that arkenfox doesn’t
https://github.com/arkenfox/user.js/wiki/1.1-to-arkenfox-or-not
> Now you can start enjoying some of the Tor Level protections arkenfox has enjoyed since Firefox 52
https://github.com/arkenfox/user.js/wiki/3.3-Overrides-%5BTo-RFP-or-Not%5D
> “One of the major difficulties Thorin experienced in her relationship with the Peacock was learning to distinguish between him pretending to be stupid just to get people off their guard, pretending to be stupid because he couldn’t be bothered to think and wanted someone else to do it for him, pretending to be outrageously stupid to hide the fact that he actually didn’t understand what was going on, and really being genuinely stupid.” – Douglas Adams
https://github.com/arkenfox/user.js/wiki/3.3-Overrides-%5BTo-RFP-or-Not%5D#-fingerprinting
> Changing a few prefs from default is not going to make you “more unique” – there is no such thing
https://github.com/arkenfox/user.js/wiki/3.3-Overrides-%5BTo-RFP-or-Not%5D#-arkenfox
> Arkenfox does not and never has, claimed to defeat advanced fingerprinting and does not care if two or three prefs with real-world tangible benefits change any stable metrics, because you are already unique
>
> Arkenfox’s primary objectives have always been security, privacy and mitigating the very real and substantial forms of tracking such as state and navigational, rather than prioritizing the potential threat of a widespread advanced fingerprinting script.
>
> That said, arkenfox does resist stateless tracking
Ignore the ignorant troll – https://github.com/arkenfox/user.js/issues/1274#issuecomment-965694513
> the hardening meme is mandatory
For the uninformed, yes. There is a reason why the Tor project speaks out against custom setups of whatever shape or form. They do not solve anything.
> For the uninformed
looks like the uninformed one is Iron Heart, the more nothing changes, the more it stays the same
hardening meme!
Anyways, brave is trash and you’ll eventually abandon it just like you did with Firefox. Firefox > brave in any regard, there’s nothing brave does better than Firefox, stay mad.
Enjoy using your clownium browser LMFAOOOOOO
Here is meme
https://github.com/arkenfox/user.js/blob/master/user.js
Thank you meme for the link. This arkenfox project is very impressive and they obviously know what they’re doing
@Iron Heart
Well, Tor Project is a really bad network, really vulnerable to sybil attacks, which means they have tons of malicious nodes which they won’t do much about, also, it is really insecure it hasn’t changed much, it still like when that dude got caught using Tor network to make bomb threats in harvard or whatever university.
But not all, the worst part is that it is a government funded tool, you know all the agencies that always fund this kind of projects.
So let me laugh a little about your post for the “uninformed” and for believing (no providing sources) the Tor project speaking about something that they probably don’t even care about because all they need (have done) is to create this fake reality where the dumb humans fall easily for the scam with the lovely and nice dream about how you can get anonymity, security, privacy and the bunch of buzz lies words with a government funded tool.
Firefox is already a terrible browser, now, Tor is built on top of a terrible browser just for the ‘agencies’ to get easy access to people’s information, so I am sure Arkenfox can’t be worst than anything you are talking about in this post, I mean, you don’t even use Firefox or maybe you do just so you can complain more about it, but you seem to care so much about the problems Arkenfox can cause, while ignoring all the BS Tor while still promoting them like “since they are against it, I should be against it, because they know what they talk about”
So, please delete post and move on.
> There is a reason why the Tor project
You do realize that this is not the Tor Browser, right? It’s very different and has little in common. In fact it tells users to use Tor Browser in many places – in the user.js, in the wiki in at least three places, in the readme at github. Arkenfox does not pretend to be something it isn’t.
Do you also realize that the benefits of using uBlock Origin far outweigh some potential costly fingerprinting proof of concept, and that arkenfox does not even claim to protect against this. And you do realize that this also applies to other browsers, right?
Do you also realize that arkenfox outside of RFP values, only changes a couple of stable and meaningful fingerprint values for a net benefit and actually warns users to not change many others because they are not necessary, including simple fingerprinting differences.
Why are you so mad at arkenfox that you have to keep making up lies about it?
Arkenfox clearly has always added privacy benefits, including fingerprinting
—
constantly repeating lies does not make them true
– https://www.ghacks.net/2021/10/21/mozilla-firefox-microsoft-store-version/#comment-4506911
– https://www.ghacks.net/2021/10/21/mozilla-firefox-microsoft-store-version/#comment-4507037
– and comments either side of these
Arkenfox is clearly not trying to be Tor Browser
The reasons?
@aaa
You need a sizable crowd to hide in, in order to beat scripts that analyze your setup comprehensively for fingerprinting purposes. Where can I find this crowd with “arkenfox”? Your numbers are totally irrelevant compared even to all Firefox installations, let alone all browsers. arkenfox also suffers from fragmentation as it is not just for ESR, but rather for the rapid release “stable” versions of Firefox as well. And no, user agent tinkering doesn’t solve this because feature detection (i.e., what web standards your version supports) is a thing.
arkenfox provides no solution for IP address tracking and some of the extension recommendations make tracking easier (e.g., uBlock Origin list config leak) rather than harder.
Enough reasons for one day?
Good to see you too Iron Heart.
Funny you said all this when Arkenfox got a wiki overhaul and things are more clear than ever including bits about Tor browser. Better read those first.
Short version: Telemetry caused it.
Even shorter: all-mighty-g they get hundred mil from, screw them from behind the cloud…
Time for mozilla’s shill army to deploy into the interwebs and execute damage control mode. Pro Tip of the day, dump Firefox!
Enjoy your botnet
@Sara
How does it feel patting yourself on the back by consciously using Firefox – Google’s vassal?