Mozilla explains the January 2022 Firefox outage that blocked connections

Martin Brinkmann
Feb 3, 2022
Firefox
|
36

On January 13, 2022, Firefox users from all over the world started to report connection issues. The browser failed to connect to any site and users were reporting hangs and crashes.

mozilla firefox stuck diagram

Mozilla published a detailed technical explanation of the incident on the company's Mozilla Hacks website on February 2, 2022.

The organization received reports about Firefox hanging during connection attempts on January 13, 2022. At the time, it saw that crash reports were spiking but did not have much information about what was causing the issue.

ADVERTISEMENT

Mozilla engineers discovered that a network request was causing the hangs for Firefox users. Engineers looked at recent changes or updates, but did not find any that could cause the issue that users experienced.

Mozilla suspected that the issue could have been caused by a recent "invisible" configuration change by one of the cloud providers that it uses for load balancing. The organization uses the infrastructure of several providers for services such as crash reporting, telemetry, updating or certificate management.

Settings were not changed in inspection, but engineers noticed that the Telemetry service was serving HTTP/3 connections, which it had not done before. HTTP/3 was disabled by Mozilla and users could finally use Firefox again to connect to services. The HTTP/3 setting at the cloud provider was configured with the automatic value.

Mozilla investigated the issue in more detail after the most pressing issue had been taken care of.  All HTTP/3 connections go through the networking stack Necko, but Rust components use a library called viaduct to call Necko.

Necko checks if a header is present and if it is not, will add it. HTTP/3 relies on the header to determine the request size. Necko checks are case-sensitive. It now happened that the requests that passed through viaduct were put into lower-case automatically by the library; this meant that any request through viaduct that added a content-length header passed Necko but ran into troubles with the HTTP/3 code.

The only Rust component that uses the network stack and adds a content-length header is the Telemetry component of the Firefox web browser. Mozilla notes that this was the reason why disabling Telemetry in Firefox resolved the issue on the user side. Disabling HTTP/3 did also resolve it.

The issue would cause an infinite loop, which blocked all further network communication because "all network requests go through one socket thread" according to Mozilla.

Mozilla states that it has learned several lessons from the issue. It is investigating all load balancers and reviewing their configurations so that future issues like that can be avoided. The deployment of HTTP/3 at Google, which was the cloud provider in question, was unannounced. Lastly, Mozilla plans to run more system tests in the future with "different HTTP versions".

Closing Words

Mozilla reacted quickly to the emergency situation and has resolved it. It may have damaged the reputation, and some users may have switched to a different browser in the process. Mozilla should ask itself whether it is a good idea to rely on cloud infrastructure that is operated by its biggest rival in the browser space. Some Firefox users may also suggest that the organization looks at the browser's handling of requests to make sure that unnecessary ones, e.g. the reporting of Telemetry or crash reporting, will never block connections the user attempts to make in the future.

Now You: what is your take on the incident?

Summary
Mozilla explains the January 2022 Firefox outage that blocked connections
Article Name
Mozilla explains the January 2022 Firefox outage that blocked connections
Description
Mozilla published detailed information about the January 2022 outage that affected Firefox installations worldwide.
Author
Publisher
Ghacks Technology News
Logo
Advertisement

Previous Post: «
Next Post: «

Comments

  1. 1337 said on February 3, 2022 at 9:14 am
    Reply

    Time for mozilla’s shill army to deploy into the interwebs and execute damage control mode. Pro Tip of the day, dump Firefox!

    1. Sara said on February 3, 2022 at 2:51 pm
      Reply

      Enjoy your botnet

      1. Dumbledalf said on February 5, 2022 at 1:30 pm
        Reply

        @Sara

        How does it feel patting yourself on the back by consciously using Firefox – Google’s vassal?

  2. Commenter McCommentface said on February 3, 2022 at 9:19 am
    Reply

    Short version: Telemetry caused it.

    1. non_commenter said on February 3, 2022 at 12:38 pm
      Reply

      Even shorter: all-mighty-g they get hundred mil from, screw them from behind the cloud…

  3. aaa said on February 3, 2022 at 9:21 am
    Reply

    Installing arkenfox.js is mandatory.

    1. Iron Heart said on February 3, 2022 at 9:24 am
      Reply

      > the hardening meme is mandatory

      For the uninformed, yes. There is a reason why the Tor project speaks out against custom setups of whatever shape or form. They do not solve anything.

      1. aaa said on February 3, 2022 at 9:34 am
        Reply

        The reasons?

      2. Iron Heart said on February 3, 2022 at 9:45 am
        Reply

        @aaa

        You need a sizable crowd to hide in, in order to beat scripts that analyze your setup comprehensively for fingerprinting purposes. Where can I find this crowd with “arkenfox”? Your numbers are totally irrelevant compared even to all Firefox installations, let alone all browsers. arkenfox also suffers from fragmentation as it is not just for ESR, but rather for the rapid release “stable” versions of Firefox as well. And no, user agent tinkering doesn’t solve this because feature detection (i.e., what web standards your version supports) is a thing.

        arkenfox provides no solution for IP address tracking and some of the extension recommendations make tracking easier (e.g., uBlock Origin list config leak) rather than harder.

        Enough reasons for one day?

      3. Yash said on February 3, 2022 at 11:11 pm
        Reply

        Good to see you too Iron Heart.
        Funny you said all this when Arkenfox got a wiki overhaul and things are more clear than ever including bits about Tor browser. Better read those first.

      4. tom said on February 3, 2022 at 12:40 pm
        Reply

        > There is a reason why the Tor project

        You do realize that this is not the Tor Browser, right? It’s very different and has little in common. In fact it tells users to use Tor Browser in many places – in the user.js, in the wiki in at least three places, in the readme at github. Arkenfox does not pretend to be something it isn’t.

        Do you also realize that the benefits of using uBlock Origin far outweigh some potential costly fingerprinting proof of concept, and that arkenfox does not even claim to protect against this. And you do realize that this also applies to other browsers, right?

        Do you also realize that arkenfox outside of RFP values, only changes a couple of stable and meaningful fingerprint values for a net benefit and actually warns users to not change many others because they are not necessary, including simple fingerprinting differences.

        Why are you so mad at arkenfox that you have to keep making up lies about it?

        Arkenfox clearly has always added privacy benefits, including fingerprinting

        constantly repeating lies does not make them true

        https://www.ghacks.net/2021/10/21/mozilla-firefox-microsoft-store-version/#comment-4506911
        https://www.ghacks.net/2021/10/21/mozilla-firefox-microsoft-store-version/#comment-4507037
        – and comments either side of these

        Arkenfox is clearly not trying to be Tor Browser

      5. Dorian said on February 3, 2022 at 12:54 pm
        Reply

        @Iron Heart
        Well, Tor Project is a really bad network, really vulnerable to sybil attacks, which means they have tons of malicious nodes which they won’t do much about, also, it is really insecure it hasn’t changed much, it still like when that dude got caught using Tor network to make bomb threats in harvard or whatever university.

        But not all, the worst part is that it is a government funded tool, you know all the agencies that always fund this kind of projects.

        So let me laugh a little about your post for the “uninformed” and for believing (no providing sources) the Tor project speaking about something that they probably don’t even care about because all they need (have done) is to create this fake reality where the dumb humans fall easily for the scam with the lovely and nice dream about how you can get anonymity, security, privacy and the bunch of buzz lies words with a government funded tool.

        Firefox is already a terrible browser, now, Tor is built on top of a terrible browser just for the ‘agencies’ to get easy access to people’s information, so I am sure Arkenfox can’t be worst than anything you are talking about in this post, I mean, you don’t even use Firefox or maybe you do just so you can complain more about it, but you seem to care so much about the problems Arkenfox can cause, while ignoring all the BS Tor while still promoting them like “since they are against it, I should be against it, because they know what they talk about”

        So, please delete post and move on.

      6. meme said on February 3, 2022 at 2:26 pm
        Reply
      7. Davey said on February 3, 2022 at 4:02 pm
        Reply

        Thank you meme for the link. This arkenfox project is very impressive and they obviously know what they’re doing

      8. :-) said on February 3, 2022 at 6:00 pm
        Reply

        Anyways, brave is trash and you’ll eventually abandon it just like you did with Firefox. Firefox > brave in any regard, there’s nothing brave does better than Firefox, stay mad.

        Enjoy using your clownium browser LMFAOOOOOO

      9. hardening meme! said on February 3, 2022 at 9:23 pm
        Reply

        hardening meme!

      10. winnie_the_pooh said on February 4, 2022 at 3:41 pm
        Reply

        > For the uninformed

        looks like the uninformed one is Iron Heart, the more nothing changes, the more it stays the same

    2. Peacock 365 said on February 3, 2022 at 10:05 am
      Reply

      https://github.com/arkenfox/user.js/wiki/1.1-to-arkenfox-or-not

      > Now you can start enjoying some of the Tor Level protections arkenfox has enjoyed since Firefox 52

      https://github.com/arkenfox/user.js/wiki/3.3-Overrides-%5BTo-RFP-or-Not%5D
      > “One of the major difficulties Thorin experienced in her relationship with the Peacock was learning to distinguish between him pretending to be stupid just to get people off their guard, pretending to be stupid because he couldn’t be bothered to think and wanted someone else to do it for him, pretending to be outrageously stupid to hide the fact that he actually didn’t understand what was going on, and really being genuinely stupid.” – Douglas Adams

      https://github.com/arkenfox/user.js/wiki/3.3-Overrides-%5BTo-RFP-or-Not%5D#-fingerprinting

      > Changing a few prefs from default is not going to make you “more unique” – there is no such thing

      https://github.com/arkenfox/user.js/wiki/3.3-Overrides-%5BTo-RFP-or-Not%5D#-arkenfox

      > Arkenfox does not and never has, claimed to defeat advanced fingerprinting and does not care if two or three prefs with real-world tangible benefits change any stable metrics, because you are already unique
      >
      > Arkenfox’s primary objectives have always been security, privacy and mitigating the very real and substantial forms of tracking such as state and navigational, rather than prioritizing the potential threat of a widespread advanced fingerprinting script.
      >
      > That said, arkenfox does resist stateless tracking

      Ignore the ignorant troll – https://github.com/arkenfox/user.js/issues/1274#issuecomment-965694513

    3. Nihal247 said on February 3, 2022 at 10:09 am
      Reply

      Yes, it is true that arkenfox increases privacy by mitigating state and navigational tracking among other methods. Seems as if Iron Heart is being very disingenuous making claims that arkenfox doesn’t

    4. bbb said on February 3, 2022 at 1:19 pm
      Reply

      @aaa I agree. arkenfox.js is brilliant

    5. arthur said on February 3, 2022 at 1:35 pm
      Reply

      arkenfox is like firefox on steroids – https://privacytests.org/ – and that doesn’t even list all the things that dFPI/FPI covers

      https://github.com/arthuredelstein/privacytests.org/issues/80#issuecomment-1008422947

    6. ULBoom said on February 3, 2022 at 5:15 pm
      Reply

      Yeah, as long as you READ what each swich does, just inserting user.js can cause issues. But the potential downsides are all well explained. Make a profile or at least prefs.js backup before experimenting and get ready for lots of reading.

      Gives more privacy and obliterates all the ad junk FF and other browsers drown in.

      LibreFox is a much simpler way for newbies to get to about the same place; arkenfox is more complete.

      1. Roadrunner 999 said on February 3, 2022 at 5:56 pm
        Reply

        > Yeah, as long as you READ what each swich does

        Stop being a baby – https://github.com/arkenfox/user.js/wiki/3.2-Overrides-%5BCommon%5D – pick and change the things you need/want

        > These few items, out of 140+ pref changes, will solve 99% of usability issues and you only need to do it once

  4. kidding_censor-hacks said on February 3, 2022 at 12:32 pm
    Reply

    “Mozilla should ask itself whether it is a good idea to rely on cloud infrastructure that is operated by its biggest rival in the browser space.”

    Are U kidding?? Currently they get $400M+ a year for only including all-G-search!!

  5. Mike said on February 3, 2022 at 1:45 pm
    Reply

    This reminds me of the addon-ocolyps, which actually wasn’t that long ago.

    https://www.ghacks.net/2019/05/04/your-firefox-extensions-are-all-disabled-thats-a-bug/

    I wish Firefox was more like Blender, a program that doesn’t do random stuff behind my back, and doesn’t randomly break for no reason.

  6. Honorius said on February 3, 2022 at 2:08 pm
    Reply

    > Mozilla notes that this was the reason why disabling Telemetry in Firefox resolved the issue on the user side. Disabling HTTP/3 did also resolve it.

    The fact that Mozilla developers and testers didn’t discover this problem BEFORE the release confirms that Mozilla employees themselves turn off telemetry (otherwise the problem would have been discovered before the release by themselves).
    What an irony!

    1. Anonymous said on February 3, 2022 at 11:28 pm
      Reply

      Nah, they use either use Edge or Chrome not Firefox lol

  7. Peter Newton said on February 3, 2022 at 4:56 pm
    Reply

    If anyone is seriously interested in minimizing privacy issues in any browser, not bothering to turn off telemetry, despite having the option to do this in the browser, is asking for trouble. If users were to take these steps, by default practice, to reduce privacy invasive issues, then in this case, and in addition, a flaw could have been avoided as well as increasing security.

    The bottom line is, try to do your best to modify your settings, to prevent anything and everything from being used against you.

    Peter Newton [London UK]

    1. ULBoom said on February 3, 2022 at 5:21 pm
      Reply

      If they can take your data, you can do whatever you want to try and prevent it. It’s a game, users who don’t play already lost. Not just FF, any online service, OS, whatever.

  8. ULBoom said on February 3, 2022 at 5:27 pm
    Reply

    Some Google guy who works in the dark, literally, will hear about this for years.

    1. Anonymous said on February 3, 2022 at 7:05 pm
      Reply

      viaduct is an internal Moz library, nothing to do with Google. The problem started and ends with problems in the Moz software.

  9. common sense computing said on February 3, 2022 at 5:53 pm
    Reply

    In the not too distant future, most software will fail to run by design if they can’t reach ad and tracking servers.

    1. anonymous said on February 3, 2022 at 7:26 pm
      Reply

      You’re probably right, I’m afraid.

    2. Anonymous said on February 3, 2022 at 10:33 pm
      Reply

      Basically, is what Windows 11 is becoming. Better start learning Linux from now.

  10. Wagons! EAST! said on February 4, 2022 at 1:33 am
    Reply

    @ Dorian
    > Tor is built on top of a terrible browser just for the
    > ‘agencies’ to get easy access to people’s information

    You are free to use another browser with Tor you know. Sure, “Tor Browser” has FF but even then, so long as you have Tor Browser running, you can configure and use another browser. You can also use Tor without FF, by installing it yourself and not using the Tor Browser Bundle. There is also TAILS Linux.

  11. dlarryB said on February 13, 2022 at 7:31 am
    Reply

    I can no longer get Firefox to work. I have uninstalled and reinstalled several times.

Leave a Reply

Check the box to consent to your data being stored in line with the guidelines set out in our privacy policy

We love comments and welcome thoughtful and civilized discussion. Rudeness and personal attacks will not be tolerated. Please stay on-topic.
Please note that your comment may not appear immediately after you post it.