A solution to ETAg tracking in Firefox

Martin Brinkmann
Dec 9, 2017
Updated • Dec 9, 2017
Firefox
|
111

The ETAg -- entity tag -- is a web cache validation method that web servers use for identifying resources. The core idea behind the feature is to use it to compare resources to determine whether they are identical or not.

As is the case with many Web features nowadays, they can be used for good and bad. ETAgs are used in the HTTP header which means that they can be used even if the browser rejects JavaScript, cookies or local storage.

Tip: We talked about ETAg tracking back in 2014, and mentioned it back in 2010 in the Evercookie article as well.

Back in 2011, researchers at UC Berkely discovered that websites were using ETAgs for tracking purposes. ETAgs are cached by the browser, and returned by the browser to the web server when a resource is requested again. The use of ETAgs allowed sites to track users across sessions, regardless of whether they changed their IP addresses, allowed cookies and JavaScript, allowed the storing of content on the local system, or had plugins enabled.

Clearing the web browser cache should remove ETAgs.  Pants, who created the Ghacks user.js file, discovered some time ago that this was no longer the case in Firefox. She noticed that Firefox was not deleting ETAg data anymore when she cleared the cache of the browser, something that Firefox did before that time.

She uses memory only caching on her system, and found out that disabling both caches (memory and disk) would defeat ETAgs but that it had other consequences at the same time.

Earthling, another bright mind behind the Ghacks user.js file, found a better solution. Since ETAgs are set in headers, manipulating headers responsible will do the trick.

  1. You do need to download and install the Header Editor extension that is available on Mozilla AMO for that though.
  2. Once you have it installed, click on the extension's icon to open the editor.
  3. Click on Add to add a new rule, and fill out the following fields:
  4. Name: ETAg Removal
  5. Rule Type: Modify the response header (this changes the fields).
  6. Execute type: normal
  7. Header name: etag
  8. Click on the Save button to save the new rule.

You can test this on the cookieless cookies site to test this (with and without the header manipulation).

Note that this bug is specific to Firefox. It may also be an issue in Firefox-based browsers.

Closing Words

It is unclear when Firefox stopped removing ETAgs when clearing the browser cache, only that this is the status quo right now. A bug listing on Bugzilla@Mozilla that was created 14 years ago highlights the tracking issue associated with ETAgs.

Summary
A solution to ETAg tracking in Firefox
Article Name
A solution to ETAg tracking in Firefox
Description
The ETAg -- entity tag -- is a web cache validation method that web servers use for identifying resources that can be abused for web tracking.
Author
Publisher
Ghacks Technology News
Logo
Advertisement

Tutorials & Tips


Previous Post: «
Next Post: «

Comments

  1. antitheory said on September 13, 2019 at 2:28 am
    Reply

    Date/If-None-Modified can be used by trackers too…

  2. John Doe said on May 28, 2018 at 8:10 am
    Reply

    I mean, useless because it doesn’t work with private browsing mode.

  3. John Doe said on May 28, 2018 at 8:09 am
    Reply

    Useless addon for those who always use Firefox in private browsing mode.

  4. Anonymous said on May 6, 2018 at 4:32 am
    Reply

    https://github.com/FirefoxBar/HeaderEditor/wiki/Basic-description#other-special-features
    “When using the “modify request header” or “modify response header”,
    set the header content to _header_editor_remove_ will remove this header (valid since 3.0.5)”

    Header Editor 3.0.5 released 2018-Jan-29.
    The rule update must I, hmm?

  5. Arcionquad said on January 25, 2018 at 1:22 am
    Reply

    Classic Cache Killer 2.0.1 (a Chrome extension) stops ETag tracking on Firefox. Chrome Store Foxified sets it up as a temporary extension, which means I have to enable it with each new browsing session using about:debugging.

  6. Rick A. said on December 15, 2017 at 6:20 am
    Reply

    i Use Firefox 57.0.2, uBlock Origin, Privacy Badger, CanvasBlocker, Decentrtraleyes, Cached Web Content is Override and Limited to 0, and on http://lucb1e.com/rp/cookielesscookies/ when i type text in the box but don’t store it, then close the tab and then undo close tab the text is still there, but when i store the text then close the tab and undo the tab the text isn’t there. Closing the browser shouldn’t have anything to do with it but maybe i’m wrong. i think i’ll wait for any replies or any new comments before i install this extension and set it up as i don’t know if it’s worth it. There’s some comments saying it’s not an issue and Clearing Cache clears the ETAg’s, some comments saying otherwise.So, i’ll wait.

    1. Pants said on December 15, 2017 at 8:59 am
      Reply

      Clearing cache clears etags – the PoC site test is a little flawed (drove me nuts) and led me to think cache wasn’t cleared properly. Thanks to Kane (and others) in the first comments. The theory however, is still sound.

      So you have at least three options (and don’t forget that no-one really knows how big the threat is in the real world, and that etags have a legit purpose)
      – 1. disable both disk AND memory cache (i am not an expert, I think this means every page is a full reload, including back-tabbing your history, but there are still some paged items in memory, not sure if that makes a difference)
      – 2. clear cache on a regular basis, eg on close, and/or use an extension uMatrix has this, eg every 15 minutes – this would mean as you browse a site you could take advantage of cached assets/items when in those 15minute time-windows.
      – 3. use cache, block the etags response headers = this means, if i understand it correctly, that you will use the cached version of items unless you do a hard refresh

      Option1 – sucks a bit in terms of speed/downloading. Option2=you could be tracked over that 15+min window but its a good middle ground Option3=no tracking, no fuss, but may make you slightly more fingerprintable (again, no one knows how much it is being used as a data point)

      Option 2 = you look like 95%? of users on the internet because you send info about expiry dates (if-none-match?).
      Option 3 = you would differ from the 95% of FF users who do not block etags or cache fully disabled – this could be used as a FP’ing measurement. An etag that is designed to TRACK you WILL track you. Again, how liekly is this to happen? Who knows. An etag could be unique to you, or you could just ALWAYS hide in a small 5% (or whatever the figure may be) – so etag entropy could be unique or it could be as low as a say one in twenty.

      Did any of that ^^ help?

      1. Rick A. said on December 15, 2017 at 11:55 pm
        Reply

        @Pants – “Did any of that ^^ help?” – A little, lol. But i don’t use Cache, as i stated my “Cached Web Content is Set to Override and Limited to 0”, so i’m good right? Or is there another Cache i should worry about ?

        What would YOUR recommendation be for my stated setup? Should i install the mentioned extension in this article and set it up the way the article states? Or i’m good because of my “Cached Web Content is Set to Override and Limited to 0”?

        Thank You in advance Pants.

  7. The Dude said on December 13, 2017 at 2:16 pm
    Reply

    Martin, since my comments are being deleted I won’t post here anymore.
    Not jokes, not serious things.

    1. Martin Brinkmann said on December 13, 2017 at 3:22 pm
      Reply

      I did not delete any comment. Which comment are you talking about?

  8. b said on December 11, 2017 at 12:32 pm
    Reply
    1. Anonymous said on December 12, 2017 at 1:14 pm
      Reply

      So does Firefox

  9. Anonymous said on December 10, 2017 at 11:13 pm
    Reply

    Could this be related to Firefox saving more in the startupCache folder files starting a couple of versions ago? Does deleting those files address this?

  10. Mike said on December 10, 2017 at 6:22 pm
    Reply

    @b, not a good plan, you’re requesting resources more often = shooting yourself in the foot. At least allow memory cache. Disk cache isn’t bad either but if you have memory to spare you can do without disk cache if you don’t want persistence after reboot. I use disk cache of 250MB because I only have 4GB RAM (effectively 3.4) and these modern browsers are memory hogs.

    1. b said on December 10, 2017 at 7:24 pm
      Reply

      @Mike, thanks for your advice. appreciate. I reset memory cache to enabled but will give disk cache disabled a shot.

      1. b said on December 11, 2017 at 11:39 am
        Reply

        @Mike
        I did indeed shoot myself in the foot: disk.cache back to normal=enabled. A fine example of “learning by doing” ;-)

  11. Paul8 said on December 10, 2017 at 5:13 pm
    Reply

    [@Pants December 10, 2017 at 5:54 am #]:
    > do you enable/disable any usersettings in about:config
    “Just a few …”

    ________________________

    … yes, a mere 2,000+ tweaks !

    appreciate the work that went into that — but us average PC users find it quite intimidating to apply.

    Yes, one can pick & choose only the tweaks one likes — but that assumes a higher level of technical knowledge than I have or want.

    Is there a much shorter list of “critical” Config tweaks (Top Ten ?) that us dummies could more easily apply ??

    1. Pants said on December 10, 2017 at 11:33 pm
      Reply

      I know, it looks daunting. But excluding the deprecated section (which would add a few items for ESR users), there are only 402 active prefs (i.e not commented out)

      Users can add this to a new profile for testing, and troubleshoot by searching for [warning] and [setup]. I’ve been planning for a long time to provide a relaxed/lite branch where the difference to be honest is only about 20 prefs which covers most breakage (eg FPI (first party isolation), RFP (resist Fingerprinting), cookies, workers, OSCP, video GMP/Widevine – struggling to think of anything now for web content .. and maybe some stuff on close & clear-all-history, locationbar settings, and allow form & search history for user UI/use. The thing is, with the current settings you won’t lose anything (just check the clear on close first – eg all history is wiped) – just flip a preference if you have a problem

      1. Anonymous said on December 12, 2017 at 1:03 am
        Reply

        @Pants: The Torture Never Stop ;)

        https://bugzilla.mozilla.org/show_bug.cgi?id=1416044
        Quote from Justin Dolske: “userChrome.css […] a timebomb for users”.

    2. b said on December 10, 2017 at 5:47 pm
      Reply

      @Paul8 & @Pants
      I was the person behind the question; but a question not properly asked: I intended to ask only for settings that dealt with cache memory and the implications of altering. in stead it seems like I asked for settings overall. i am a dumb fool but not as much as a year ago ;-) apart. I dont think theres an easy way around. take your time. read in small doses.

  12. b said on December 10, 2017 at 4:33 pm
    Reply

    I decided to set browser.cache.memory.enable & browser.cache.disk.enable to false( on pale moon ). no text stored when I test cookieless cookies site. all though, as mentioned by others: it keeps showing the number 2, no matter the actual number of visits. this will probably do for now. I will however look into bleachbit.@Marti Martz: thanks for mentioning.

  13. Yuliya said on December 10, 2017 at 10:26 am
    Reply

    >Clearing the web browser cache should remove ETAgs.
    Here on ESR52 they are removed by cleaning “Form & Search History” found under “Clear Recent History”.

    I’ve been considering a change in my browsing habbits for a while: a browser (likely FireFox) always in private mode for general browsing and another browser to have all my cookies with logins, etc.. The way I do it right now is only with Fx – my normal session has all my cookies, etc.. I’d like the browser with cookies to be Fx portable, but I haven’t figured it out yet how to run both Fx portable and the normal installed one at the same time.

    1. Anonymous said on December 11, 2017 at 1:45 am
      Reply

      Note: If you really want to use Firefox Portable alongside Firefox, you can use the -no-remote flag which is supported by the portable version.

      For that you need a FirefoxPortable.ini file in Firefox Portable’s root directory. You can find one in the directory “\Other\Source”. Just copy it over to the root directory. The setting to modify is

      AllowMultipleInstances=true

    2. Anonymous said on December 11, 2017 at 1:40 am
      Reply

      @Yuliya

      Instead of two different browsers you could use two different profiles, that’s simpler IMO.

      You can run as many Firefox profiles concurrently as you want, each one with their own desktop shortcut, with “firefox.exe -P YourProfileName -no-remote”

      If you omit the profile name after the “-P” a small window will open letting you pick which profile to start, create new profiles or manage existing ones

    3. Sophie said on December 10, 2017 at 12:53 pm
      Reply

      Not sure why you might feel you have to have regular Firefox and Portable at the same time. I run Firefox 43x / Firefox 51x and Waterfox 54x …………………all as portable versions, and all at the same time. No issues, and all play nicely together.

      Also, this makes it very easy to move complete copies of Firefox to other PCs, in order to have duplicate installs/addons/settings, elsewhere.

    4. Pants said on December 10, 2017 at 11:54 am
      Reply

      I have form & search history disabled. I also have clearing form & search history checked when I manually clean, and also on close. My manual clearing is always time range “everything”, and the dialog is “Clear ALL history”. In fact I have every single option checked except cookies (and the test site cannot load a cookie because I block that) This did not clear the etag in my tests.

      As I said in my tests, I have zero persistent data. No site preferences, no cookies, no localStorage, no IDB, no appcache, no offline stuff, no service workers cache, no flash, no disk cache (but i do have memory cache), no browsing history, no form or search history — absolutely no histories at all — no windows.name (new tab, new browser session) .. I can think of nothing I didn’t block … I blocked third party, I blocked JS, and I even blocked XHR and more even though it is not there (from my uMatrix default). There isn’t even a referer.

      All I allowed was css and the single image. By changing ONLY memory cache, the test fails. By ONLY changing the etag response header, the test fails. This indicates that 1, clearing cache doesn’t clear ram and 2. etags are the cause

      It also seems changing your IP also fails the test, and the test in this regard seems flawed – by removing IP from the test, the PoC still works. As to the use of the IP and apparently UA, it seems this is used to seed a unique etag, but it will someone with a little more knowledge to delve into it. I for one do not need to waste any more time on it – I KNOW etags can be used on their own to track, and now I have a solution other than disabling both disk and memory cache

      > I’ve been considering a change in my browsing

      Do it man .. multiple concurrent profiles, multiple browsers. I do this. Don’t buy into the private window mode BS. Don’t even buy into the containers option either. Use FPI in your main FF, end of story. You could use containers on a secondary profile, eg to split some common third parties.

      I run my main FF. I use it for around 300+ bookmarked sites, they all work beautifully (except maybe some videos – I am not a video watching person anyway). I block all cookies bar 8 sites (for some logins) and another 8 sites have allow cookies for session, because they will not quite function without localstorage. Almost every other site I visit in terms of research or whatever, I can usually get a readable format (especially if in uBo I can spot the offending css block). Even if it’s not formatted and styled nicely, its enough to read it.

      Rarely I throw a site into a portable Opera, or a portable chrome. The chrome is locked down fairly well, the opera one not so much. So… FF->Chrome->Opera->can’t-be-that-important->drink-some-beersies

  14. TelV said on December 10, 2017 at 8:36 am
    Reply

    Well I can’t speak for the utilities that you use since I don’t use them myself, but by using the add-on I mentioned the cookieless site records just two visits now whereas before amending the header, it had reached eight. The additional visits had been recorded when I was wasn’t sure which menu option to use.

    Also, I’ve noticed that visiting the site directly rather than clicking the link to it in Martin’s article produces a different reading for some unknown reason.

  15. Jonnyredhead said on December 10, 2017 at 3:55 am
    Reply

    And for Opera ?

    1. Marti Martz said on December 10, 2017 at 4:10 am
      Reply

      > for Opera ?

      The one my distro has of Opera also has the issue of having the stored text save between cleaned sessions.

      The newer versions use the back end of Chrome as seen by their useragent of `Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36 OPR/49.0.2725.34` e.g. “Blink”.

  16. ULBoom said on December 10, 2017 at 3:03 am
    Reply

    So, are Etags the huge list of cookies CCleaner shows for firefox and chrome after some time spent browsing that don’t disappear with cache flushes? I can’t find those cookies anywhere in ff’s or chrome’s program files or with a search unless they’re hidden in js files or somewhere else. The chrome I use is the stripped woolyss chromium.
    Also, a thanks to those who ferret out all these things!

    1. Pants said on December 10, 2017 at 6:26 am
      Reply

      No, they are entries in your profile’s SiteSecurityServiceState.txt file – CCleaner adds these as cookies under it’s analysis. You can blank the file and change its write permission if you wish. This is used for HSTS stuff

      1. Sophie said on December 11, 2017 at 8:45 am
        Reply

        @pants – is this completely correct and true? If so – this answers a question I have had for ages…..namely….why does CCleaner show lots of “cookies”, when cookies are cleared down and removed (or mostly).

        So, they are coming from HSTS! I am happy to know this. I have a batch file I set up to copy over a blank HSTS text file over the top of the populated one, each time Firefox restarts or on a re-boot. For this reason, very few entries of displayed cookies show up on CCleaner, but some still do…if the file has not been cleared down.

        What you have said seems to explain where they come from, which was always a mystery to me, same as ULBoom has described.

  17. Peter said on December 9, 2017 at 10:51 pm
    Reply

    Pale Moon 27.6.X clears them if “site connectivity data” is checked in “clear recent history” (ctrl-shift-del) menu
    No addon needed.

    1. b said on December 10, 2017 at 11:34 am
      Reply

      not for me. been using ctrl-shift-del for a while; but it does not solve the problem. i run ubuntu. don’t know if that explains it

  18. Sampei Nihira said on December 9, 2017 at 9:08 pm
    Reply

    https://anonymous-proxy-servers.net/blog/index.php?/archives/400-Cookieless-Cookie-Fake-Test.html

    …..Ok – after some more test and logging the HTTP header it was clear. The test uses the IP address for session tracking and the user agent but not ETags. If an ETag was sent or not didn’t affect the test result. The claims of the author are lies, it is a fake test…..

    1. Pants said on December 10, 2017 at 7:40 am
      Reply

      OK, I have digested that link a bit now. First, I do not dispute what cane wrote, those are his actual results. I will ignore all the stuff about SDC – I do not use it and never have. The post is over 4 years old.

      Since then, the clearing cache (manual or on close) no longer works – as per the article above. It used to, no idea when that happened. So JoDonym is in the same boat as all of us since they use Firefox. Here is a pic from around FF38 when it did work – https://user-images.githubusercontent.com/16656956/28538260-c1385a50-7101-11e7-9194-2a3a99243286.png . That JoDonym test part no longer exists. I would test it by clearing cache on the test link page – http://ip-check.info/?lang=en .

      All I can surmise from the IP changing (and no clearing cache or restart in between) is that maybe some items are session’ed by Firefox by IP and that changing IP also clears your etags – as evidenced by his first test and not getting a 5th visit indicator

      1. claustromaniac said on December 11, 2017 at 8:36 am
        Reply

        To further clarify (I’m not particularly good at this kind of explanations):

        Say you visit the test site for the first time.

        1) Your browser initiates the first request.
        2) Server looks for If-None-Match header, and since it doesn’t find it, it generates a new unique ETag to track you (something like… 6185-4e427532a9640) , and sends it back along with the image.
        3) Your browser caches the image and stores the ETag.
        4) You reload the page to see if your individual text string and number of visits are still OK, and they are.

        You then change your IP and reload the page again. This is what *should* happen:

        5) Your browser initiates request, sending along If-None-Match: 6185-4e427532a9640 (the previous ETag).
        6) Server looks for a matching session in its database, which should be there.
        7) You reload once again and you should then see appropriate number of visits and such.

        As to what is really happening in steps 5 and above… your guess is as good as mine. The only thing I can tell you with relative certainty is that the IP and user-agent string are only used to generate the ETag in the code, and nothing else.

      2. claustromaniac said on December 11, 2017 at 5:52 am
        Reply

        It looks like I’m kind of late to the party, but I will try to elaborate on what I pointed out at GitHub.

        If I understand correctly, the whole thing about the test using the IP and user-agent to generate the ETag just means the test uses a lazy (but theoretically effective) method to give each user a unique ETag. This is a fundamental part of using ETags for tracking: if the cached images for two or more different users share the same ETag, the test is pointless, because you can no longer tell them apart from each other.

        The test could instead use other methods for that, but it would be pretty hard to find another effective method as simple as that one. The whole ETag is generated in a single line of code with this method, whereas, for example, generating a random string of text and comparing with previous sessions would require way more than that, and would likely slow down the test to a crawl.

        Once the ETag is generated and saved, it shouldn’t matter whether the user’s IP changes or not, because the test only uses the ETag that the user’s browser is sending back to track them.

        That’s why I don’t understand why changing your IP affects the result in reality. But I repeat: I’m by no means an expert on this whole subject. I might be missing something.

      3. Pants said on December 10, 2017 at 11:20 am
        Reply

        @sophie .. calm down :) First of all the technicality of tracking via ETags is well known, even if the test site’s PoC is flawed (not fake, maybe flawed in some cases). Secondly, the solution, albeit not perfect because it also blocks legitimate etags, DOES work. It’s a no brainer – a site can tag you with a unique etag (how it generates that is up to them), and the blocking of etag response headers breaks that tracking ability, AND, it takes nothing else to read back an etag, end of story. This focusing on the IP/UA etc is a red herring .. an interesting herring, but still a red one

        Back to that test. Seems the site (and I am not an expert) **may** be using the IP & UA to **seed** an initial etag (which seems a bit silly when you think about it), in which case changing the IP means the test breaks (which is why I said to eliminate the ip changes in your test). Whatever is used to seed the etag (in this case ip & ua?) does not invalidate the test (if you eliminate ip & ua in your test).

        Maybe someone else with a little more knowledge can work out what is going on “exactly”. The whole thing is open sourced on github, and the actual test page speaks about bugs and shortcomings of the test etc if anyone cared to read them

      4. Sophie said on December 10, 2017 at 9:54 am
        Reply

        What a fuck! You want to tell me, you can track a JonDoFox by using ETags? Bullshit! Let’s make an moose accident test and change the IP address. The author claims, that he don’t use the IP address but only “ETags from browser cache. I switched the mix cascade and reload the test with a new IP address, without browser restart and without clearing the cache. If it was possible for the test to track my browser I have to see “5” visits and my text, but I got:
        Number of visits: 2
        Stored text: ”

        Ok, so back to Pants…and the reply to me above (@sophie) …. it “does” seem to me that the test uses IP address, and therefore fails to track if this changes.

        To me, this negates the need to worry about this, and supports the possibility that the cookieless-cookie test is at least misleading, if not an outright fake.

    2. Pants said on December 10, 2017 at 7:17 am
      Reply

      Digesting your link a bit more. The methodology of tracking is definitely possible (as outlined back in 2000 etc), and the modifying etag response header fixes that – its not a perfect fix, agreed. As in the github issue, we struggled to find tests, and even questioned this PoC a bit (sometimes results were dodgy)

      Thanks for the JoDonym link.

    3. Pants said on December 10, 2017 at 7:02 am
      Reply

      Are you sure? How come I can break the tracking when I do not change my IP or UA, and the ONLY change between tests is to modify the ETag response header

      1. Kane said on December 11, 2017 at 1:18 am
        Reply

        Just tested it, clearing just the cache and changing useragent is enough to get the server to send a different ETag. No need to go through the trouble of changing IP, so many people are stuck with a static IP these days…

      2. Kane said on December 11, 2017 at 1:06 am
        Reply

        Because it’s the response header that you change, in other words the data freshly coming from the server.

        The server received your IP and useragent, made an ETag from at least the IP and sent it back to you, and instantly your add-on modified that ETag to an empty value. That’s what your browser will see, but the server did send the same ETag.

        Then upon reload, your browser will not send an ETag in the request headers (no If-None-Match field), but the server will send the same ETag again in the response, which will be emptied again. Now without the add-on, if you deleted the cache before reloading the same thing would have happened, the only difference being that the response wouldn’t be emptied upon reception from the server. From a tracking point of view the add-on is not providing a protection but arguably increasing your fingerprint entropy. (See my other comments regarding this. It’s starting to be all over the place O_o)

        So ETags are cleared with cache. To get the server to send a different ETag, one needs to change IP. I haven’t tried it, but changing the useragent could work too, depending on how the ETag is set server side. (Probably just uses IP though)

        There’s a set of steps to reproduce this in one of my other comments.

  19. Uninstalled said on December 9, 2017 at 8:13 pm
    Reply

    Strange. Downloaded Header Editor and applied rule and saved it. When checking it again there was no rule showing at all. Guess it did not work or it does not show anything by default. Uninstalled, since I was unable to figure out what it is doing and what not.

  20. TelV said on December 9, 2017 at 7:50 pm
    Reply

    Since the add-on didn’t work for me on Basilisk I had a hunt around for another one which is called Modify Headers: https://addons.mozilla.org/en-US/firefox/addon/modify-headers/

    It works in a similar way and I took a screenshot of it after testing it thoroughly: https://imgbox.com/bUj2rxbM

    After you install it click the “Select Action” dropdown and choose “Filter” from the menu. Type: Remove Etag and click “Add” and then click OK to exit. I didn’t bother with any other description although there is a field available for that.

    But it doesn’t work on FF57+ I regret to say.

  21. Sophie said on December 9, 2017 at 7:48 pm
    Reply

    Unless I’m missing something here, I find it hard to see this as a problem. In my case, I delete most cache elements on each browser restart. But I don’t delete cookies.

    What I found, as I believe Albion Rammsey was saying too…..is that (with a cleared cache, and a different IP via Proxy or VPN), then the “persistence” is gone: it appears as my first visit, with only the latest (current) time stamp.

    So really, this is not tracking at all? If a cleared cache and a change of IP address is all it takes to clear it down and make visits look like the first, then surely it fails in its’ ability to track. If so, then Header Editor Addon is not needed, nor anything else in such a scenario.

    Am I right?

    1. Pants said on December 10, 2017 at 6:24 am
      Reply

      “with a cleared cache” – Firefox is not clearing this fully, ETags persist

      “ip change” – remove this from the test to get proper results. This variable is not needed for the PoC or “fix”

      “this is not tracking at all” – umm, yes it is, as proven by the PoC

      “If a cleared cache and a change of IP address is all it takes” – you have to use a third party to clear your cache, and this may not be practical. Additonally, the IP issue is conflating the threat model. Even if the test site was using IP (which is not AFAIK), then this is not practical either – not everyone will be flicking their IP every 10 minutes such as with JoDonym (but yes, this is a good measure if you can do it)

      How big is this tracking threat – who knows. Etags serve a purpose, but can be misused, as outlined way back in 2000 – see http://sourcefrog.net/projects/meantime/ (scroll down to the “ETag and Last-Modified headers” section) and as filed under bugzilla 14 years go – https://bugzilla.mozilla.org/show_bug.cgi?id=231852

      It would be be to get some real world metrics eg from say analysis on the top 1million Alexa ranked sites

      1. Sophie said on December 10, 2017 at 9:45 am
        Reply

        Thank you Pants, for your helpful answer!

  22. Arcionquad said on December 9, 2017 at 7:29 pm
    Reply

    As Marti Martz has already said, Chrome also stores Etags, even after closing the browser. The Header Editor extension is also in the Chrome Web Store, and it works — just like it does in Firefox.

    1. Pants said on December 10, 2017 at 6:15 am
      Reply

      There is a difference between how Chrome and FF handle “removing” headers – from the developer [1]

      “Chrome and Firefox is different when you try to modify a header to blank. Firefox will remove it, but Chrome will keep it and set it to blank”

      [1] https://github.com/FirefoxBar/HeaderEditor/issues/43#issuecomment-350416556

      1. Kane said on December 11, 2017 at 12:46 am
        Reply

        No it is terrible, you will be uniquely identified with Chrome if that quote is true.

        With Firefox you will be one of the rare users not to accept ETags… I’m going to hypothesize that it’s even rarer than refusing cookies, which IIRC is 2-3% of Firefox users.

        Don’t use the add-on, just clear your cache as well when you want to clear cookies. You could also use an add-on that auto-deletes cache, like there are add-ons that auto-delete cookies like Cookie Autodelete. uMatrix can do that if configured to.

         

        Note: I haven’t checked that Chrome deletes ETags along with the cache, but Firefox positively does. (See my other posts)

      2. Arcionquad said on December 10, 2017 at 8:15 pm
        Reply

        Thanks, Pants, for your reply, which caused me to go back and repeat the Etag tracking test in Chrome. When I cleared the cache on the Chrome browser and then refreshed the page, the text that I saved on https://lucb1e.com/rp/cookielesscookies/ disappeared — and this happened whether the rule in Header Editor for Chrome was enabled or disabled. I was wrong to assume that Header Editor was necessary to accomplish Etag removal in Chrome.

      3. Klaas Vaak said on December 10, 2017 at 10:52 am
        Reply

        So is it worth installing the Header Editor in Chrome or not?

  23. RAS said on December 9, 2017 at 7:22 pm
    Reply

    The Random Agent Spoofer extension for Firefox has an ETag spoof setting.

  24. Doc said on December 9, 2017 at 7:09 pm
    Reply

    Martin, it should be “ETag” – there’s no reason to capitalize the “A”.
    https://en.wikipedia.org/wiki/HTTP_ETag

    1. Marti Martz said on December 9, 2017 at 7:48 pm
      Reply

      > it should be “ETag” – there’s no reason to capitalize the “A”.

      Does look better but case insensitive.

      See also:
      * https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2

  25. Torro said on December 9, 2017 at 6:19 pm
    Reply

    AM using FF 57.0.2 and when i trued adding the rule, nothing happens. not sure what is wrong. removing add-on and trying again but nothing happens. Anyone else having this issue?

    1. Pants said on December 10, 2017 at 6:12 am
      Reply

      – The extension stores your rules in IDB. So one, you will need to create them in normal window mode since private browsing mode windows do not allow IDB.
      – If you are blocking all cookies by default, you will need to allow a cookie exception for the extension – see https://github.com/ghacksuserjs/ghacks-user.js/wiki/4.1.1-Setting-Extension-Permission-Exceptions
      – make sure the Header name is “etag” (without quotes) – it is case sensitive because the extension does not use toLowerCase to compare (although I logged an issue about this thanks to earthlng)

      PS: I have not tested if this work in PB Mode due to IDB restrictions

      1. Anonymous said on December 12, 2017 at 11:23 pm
        Reply

        uMatrix handles cookies too.

        It accepts them all but doesn’t save them back unless you allow. That means for instance on YouTube you can set the wide screen and different resolution and the website will remember it without the need to allow cookies. (They have been modified and can be read client side, but not sent)

        The downside should be that, I guess, a website can read the content of the cookies and then them back through another mean (XHR if allowed by uMatrix, or maybe loading a web bug)

      2. Pants said on December 12, 2017 at 5:10 pm
        Reply

        I don’t use any cookie extensions – instead I use FPI, and I block all cookies by default. I then use Firefox’s site exceptions to allow about 8 cookies, and another 8 for session only. Life is easy. This approach may not work for everyone. But once you set it up for all your major sites/bookmarks then it’s done. Of all my 350 odd speed dials, and site logins, I only needed 16 exceptions. 99% of websites are 100% readable without cookies – readable, maybe not fully 10% usable/functional.

        My first thought would be that cookies allowed by Firefox (block all, allow some site exceptions) could still be handled by the extension. Maybe you needed to add the same exception to the cookie extension, or maybe the extension doesn’t like having “moz-extension” as entries. IDK. Maybe it’s a Firefox API thing, but it just sounds so silly that this is restrictive like this.

        Maybe I’ll have a play with Cookie-AutoDelete one day when it can function with FPI and handle both localStorage and IDB by host

      3. TelV said on December 12, 2017 at 4:32 pm
        Reply

        @ Pants,

        If you’re using a cookie manager such as Self Destructing Cookies which handles cookies on a site by site basis, then configuring a cookie exception doesn’t work even if you configure one. I did that as can be seen in this screenshot https://imgbox.com/MaFaRzrM but because network.cookie.cookieBehavior = 2 in about:config it will delete whatever rules have been created in the Header Editor.

        Setting network.cookie.cookieBehavior = 1 works and leaves your rules in place, but that setting also enables the “Accept cookies from sites” in about:prefs/history menu. Accepting cookies which are going to track you wherever you go just so that you can delete Etags kind of defeats the object of the exercise IMHO.

  26. Marti Martz said on December 9, 2017 at 6:19 pm
    Reply

    Seems that another more practical solution to keep actual real caching on sites enabled would be to have a shell or cmd script that executes CCleaner or BleachBit *(equivalents of course presuming they all have a CLI)* on exit.

    Restarting a browser probably wouldn’t leave the main process so an exit would be necessary to continue on.

  27. Dave said on December 9, 2017 at 6:14 pm
    Reply

    I too did the simple testing using the websites linked to in the OP.

    The tracking was working.

    The method described in the OP worked to stop it.

    TOR is not affected by this.

    I have my cache set to a folder on my ramdrive and putting the PC to sleep and waking it also clears the tracking.

    (I have a task that runs at the logon of any user that runs a bat file that creates a directory structure on the ramdrive. My TEMP and TMP system and user variables also point to the ramdrive.)

    I’ve been wondering if a websites icon that you see both in your favorites list and on the tab left of the text could be used for tracking in a similar fashing?

    1. Marti Martz said on December 9, 2017 at 6:21 pm
      Reply

      > I’ve been wondering if a websites icon that you see both in your favorites list and on the tab left of the text could be used for tracking in a similar fashion?

      Could be. Usually those get cached so I wouldn’t see why it would be any different. Some browsers periodically recheck it regardless of cache settings too.

  28. TelV said on December 9, 2017 at 6:09 pm
    Reply

    The add-on doesn’t work in Basilisk. No response from the button at all and although the menu appears when using the Add-ons Manager/Options menu nothing happens when clicking the Save button.

    On the Cookieless site Basilisk has now been recorded twice so this particular fork is also vulnerable to the issue unfortunately.

    1. TelV said on December 11, 2017 at 5:15 pm
      Reply

      I’ve since discovered that it does work on Basilisk. After posting the problem on the developer’s thread on Github he tested it in Basilisk and as can be seen from the screenshot he posted, it worked. I subsequently tested it in a new profile and likewise, it works without any problems.

      So I guess that the initial problem was either due to a conflict with another extension, or an about:config setting somewhere. I’ll have to add all the other extensions I have to the new profile and then test it again. If it continues to function properly, then it has to be an about:config setting which is screwing things up.

    2. Marti Martz said on December 9, 2017 at 6:16 pm
      Reply

      > has now been recorded twice

      I think that is the default when visiting the first time. It always says 2 for me even after cleaning with CCleaner or BleachBit.

      1. Lucifer, Alphabet CEO said on December 10, 2017 at 6:13 pm
        Reply

        Yep, me too. It says ‘2’ when loaded for the 1st time.

  29. jern said on December 9, 2017 at 5:56 pm
    Reply

    I set both the “web content” and “application cache” to zero storage. I visited the cookieless cookies website, left some text, closed Firefox. Then I took one extra step, I shut down my modem (which I always do when leaving the internet). I returned to cookieless and they showed as a new visitor. Maybe the solution is just to turn off the modem occasionally.

  30. Tom Hawack said on December 9, 2017 at 5:38 pm
    Reply

    Martin, big thanks for this very nice article. Runs flawlessly. Another brick in the privacy defense wall.

    1. Kane said on December 11, 2017 at 12:38 am
      Reply

      I think it’s a hole if anything though.

      (See my other posts, just poking to make sure you’re notified :) )

    2. Klaas Vaak said on December 10, 2017 at 10:46 am
      Reply

      Until the next one is taken out by Big Brother.

  31. Marti Martz said on December 9, 2017 at 5:16 pm
    Reply

    > Note that this bug is specific to Firefox. It may also be an issue in Firefox-based browsers.

    Present in most recent SeaMonkey and Pale Moon unfortunately. Using BleachBit, or similar, for Linux to scrub just like CCleaner is to Windows.

    Btw tampering with headers like this can bust caching and can invoke other protection methods against site abuse. So this isn’t the best solution. I do agree that once the cache is cleared that invalidates keeping the ETag.

    Logic is missing sometimes with Moz. *sigh*

    1. Marti Martz said on December 9, 2017 at 5:57 pm
      Reply

      > Note that this bug is specific to Firefox.

      Also present on Google Chrome 63.0.3239.84 (Official Build) (64-bit) with http://lucb1e.com/rp/cookielesscookies/ Stores the text entered quite well even after cleaning with the browser back to the beginning of time. :(

  32. anon said on December 9, 2017 at 4:10 pm
    Reply

    This article is inaccurate; I tested in attempt to confirm the reported issue (ETags not cleared with cache) in preparation for filing a bug report. The issue as described does not present in Firefox 57. I have confirmed that both manually clearing the cache as well as automatically clearing on shutdown result in the ETags not being sent in the “If-None-Match” HTTP request header. I confirmed by observing the headers sent via the Developer Tools Network panel; you can test yourself with any site which sends ETag, or you can write a PHP page like follows:

    <?php
    header('ETag: TestETag');
    if($_SERVER['HTTP_IF_NONE_MATCH'])
    {
    echo 'You are currently sending the following ETag: ‘.$_SERVER[‘HTTP_IF_NONE_MATCH’].’‘;
    }
    else
    {
    echo ‘You are not currently sending an ETag via If-None-Match header. (This is your first visit, or your cache was recently cleared, or you’re blocking ETag.)’;
    }
    ?>

    By the way, I also don’t know why the author is capitalizing as “ETAg” — proper capitalization is “ETag”.

    1. Kane said on December 11, 2017 at 12:32 am
      Reply

      @anon

      You are correct. It’s the testing website that caused misunderstandings. 1/ It creates the ETag based on user IP, so even when cleared, reloading will have the server generate the exact same ETag. 2/ The website displays the value from the previous update, which can lead to misunderstandings.

      And finally the add-on merely rewrites the value received from the server so it is empty. It’s essentially the same as refusing cookies: You can either accept them and delete them later, for instance with Cookie Autodelete, or you can refuse them outright. Here instead of cookies it’s the cache that is concerned: With the add-on, the ETag is set to an empty value outright. Without the add-on, the ETag is removed with the Cache. (uMatrix can auto-delete cache down to every 15 minutes if set that way.)

      Emptying ETags outright is probably quite telling when it comes to fingerprint.

    2. Tom Hawack said on December 9, 2017 at 5:44 pm
      Reply

      I use latest Firefox 57.0.2 and testing ETAg with the site mentioned in the article proved the ETAg remained after having closed Firefox, cleaned the caches, even the cookies (all cleaned, as in spring cleaning) and restarted Firefox : my little message on the test site, together with the date-time was still there. Did the test with this article’s procedure and without cleaning the caches, without closing Firefox, the ETAg had vanished.

    3. Arcionquad said on December 9, 2017 at 5:03 pm
      Reply

      The article is accurate. I use Firefox 57, and the text I stored on cookielesscookies was still there after I closed Firefox and cleared the cache. The fix recommended by Martin Brinkmann works.

      1. anon said on December 11, 2017 at 1:45 am
        Reply

        Arcionquad (and Tom Hawack, below) …

        The site is cheating. It’s storing the information based upon IP address.

        If you open the developer tools Network panel and then navigate to the tracking image (https://lucb1e.com/rp/cookielesscookies/tracker.jpg) after having cleared the cache, you will see that your request headers do NOT send the If-None-Match with any ETag, however, the response headers will send the same ETag you had before … how do they do this? They’re storing the session based upon IP address. It’s provable that they’re getting the identifier from something other than your request headers after the cache is cleared, because the request headers don’t include the identifier. In addition, when you switch back and forth between two different IPs, you will go back and forth between two different ETags!

        How to reproduce:
        On your regular home ISP, clear your cache, then open the developer tools Networking panel and go to the URL noted in prior paragraph. Click on the row for the tracker image and a pane will open on right. Go to the request headers and you’ll see there is no If-None-Match header specifying the ETag. Then look at the response headers, and you’ll see the same ETag as before. NEXT, close the page and clear your cache. Then use your mobile phone hotspot to connect your computer over cellular data. Repeat the same steps as above, and you’ll see a different Etag. Then repeats the steps again on your home ISP and you’ll get the original ETag. Then switch back to cellular and repeat and you’ll see the cellular ETag. You can do this over and over and get the same two ETags, based upon your two different connections. In other words, it’s based upon your IP address, not your cache!

        As for why blocking the ETag might have some effect on this fake proof-of-concept for purely cache-based tracking, I would presume it’s simply because they only show the info if they can get an ETag echoed back to them after the page is automatically reloaded … I guess they are simply trying to fool people, else they’re incompetent. Whatever the case, the proof-of-concept is a fake as it is not based purely upon browser cache as advertised.

  33. Arcionquad said on December 9, 2017 at 3:51 pm
    Reply

    Thank you, Martin, for the information about Etags. I tried Earthling’s solution using Header Editor, and it worked. At first, when I was in a private window, nothing happened when I clicked on the save button. Using Header Editor in a normal window did the trick.

    1. Pants said on December 10, 2017 at 6:06 am
      Reply

      PB Mode does not support IDB (indexedDB), which the extension requires to hold your rules etc. You will have to make sure the extension has a cookie permission exception if you block all cookies (as cookies control access to localStorage (not the Extensions Storage API which is fine ) & IDB).

      In a normal window you can add/edit your rules (since IDB is supported). In PB mode I assume it **only reads** the stored rules in your relevant “moz-extension+++UUID directory” under your profile/storage/default directory – don’t quote me on this, I am just guessing.

      I haven’t actually tested the solution in PB mode (which I never use because everything it does can be done in normal mode, if not better), … suppose I should

  34. Steve Hare said on December 9, 2017 at 2:50 pm
    Reply

    I found running CCleaner between sessions cleared cache.
    I also found that closing and starting FF twice in a row cleared cache.
    I am not using PB.
    I am using W10 x64 and FF 58b10 (heavily modified the user.js)
    I will play around with this on Linux Mint 18.3

    I can’t thank the folks enough who spend their time running these things down.
    I know it’s not possible to stop tracking – either by the government or big business or hackers.
    All our modern conveniences – our operating systems, our applications, the internet, our cellphones, our cars – are all used to monitor our life patterns and to exploit and control our behavior.
    Sigh. It is a shame we have to live this way.
    But it is appreciated that we at least try to stop it and that there are folks who feel the same way and have the knowledge to help the rest of us.

    1. Pants said on December 10, 2017 at 6:00 am
      Reply

      > I found running CCleaner between sessions cleared cache

      Same – see https://github.com/ghacksuserjs/ghacks-user.js/issues/179#issuecomment-319061288

    2. Satan, CEO Google said on December 10, 2017 at 3:18 am
      Reply

      The internet turned out to be Big Brother. A global Panopticon. “The Panopticon is a type of institutional building and a system of control designed by the English philosopher and social theorist Jeremy Bentham in the late 18th century. The scheme of the design is to allow all (pan-) inmates of an institution to be observed (-opticon) by a single watchman without the inmates being able to tell whether or not they are being watched. Although it is physically impossible for the single watchman to observe all the inmates’ cells at once, the fact that the inmates cannot know when they are being watched means that they are incentivized to act as though they are being watched at all times. This scheme effectively compels the inmates to constantly control their own behavior. The name is also a reference to Panoptes from Greek mythology; he was a giant with a hundred eyes and thus was known to be a very effective watchman.”

      1. Lucifer, Alphabet CEO said on December 10, 2017 at 6:10 pm
        Reply

        + Cache on RAM, not on disk.
        + CCleaner after closing the browser.

  35. Albion Rammsey said on December 9, 2017 at 2:04 pm
    Reply

    ” You can test this on the cookieless cookies site”

    I installed “Header Editor” and added the rule. Then visited the test site and then closed Firefox and used CCleaner (even though I was using private browsing mode). Restarted Firefox and went to test site and it recognised I’d been there before. I then rebooted my PC and also changed my IP. When I went to the test site it said that was my first visit.

    So yes, the Header Editor rule works, but you still have to ensure you are using a different IP address.

    Thanks.

    1. Pants said on December 10, 2017 at 5:54 am
      Reply

      CCleaner defeats the ETag and invalidates the test – see issue 179 – namely this comment https://github.com/ghacksuserjs/ghacks-user.js/issues/179#issuecomment-319061288

    2. ams said on December 9, 2017 at 10:31 pm
      Reply

      “correlation does not imply causation”

      Sorry, I’m skeptical of the “testing” regimen you’ve outlined. If changing IP address was the determining factor… how does that support your conclusion that the tracking method involved was eTag?

      1. Pants said on December 10, 2017 at 6:00 am
        Reply

        edit: I meant I did it all WITHOUT changing my IP

      2. Pants said on December 10, 2017 at 5:59 am
        Reply

        I did all my tests with changing IP – also nothing was allowed to run such as XSS, third party connections, no JS, no cookies, no storage – nothing etc. Sometimes the test was a bit flaky, but both PB mode and normal windows and across FF restarts (with both manual and on-close clearing of cache) all showed tracking could be done – essentially it came down to FF not clearing the ETag property somewhere and since AFAIK nothing was on disk, I can only assume it is still in memory, and as turning off memory cache defeated it, cogito ergo sum

  36. b said on December 9, 2017 at 1:08 pm
    Reply

    @pants
    “She uses memory only caching on her system, and found out that disabling both caches (memory and disk) would defeat ETAgs but that it had other consequences at the same time.”

    do you enable/disable any usersettings in about:config? what are the consequences that you experience, mentioned by Martin Brinkmann ? I use pale moon and waterfox these days. they both share some settings with firefox. pale moon does not support web extensions how ever, so I guess the only solution would be changing usersettings. I’m tracked on both browsers according to cookieless cookies. thought ctrl+shift+delete after closing every tab would do the job; but no

    1. Pants said on December 10, 2017 at 5:54 am
      Reply

      > do you enable/disable any usersettings in about:config

      Just a few .. see https://github.com/ghacksuserjs/ghacks-user.js/blob/master/user.js

      1. b said on December 10, 2017 at 11:22 am
        Reply

        I do actually. I’ve set the most obvious along the way ( thanks to this blog and your site ) but only recently I started a systematic reading of your suggestions. thanks to your warnings & explanations I only choose those that makes sense to my needs and that I truly understand. so far I reached the section about screen sharing. however I don’t understand the section about cache. what exactly are the problems, I might enter if I change my settings in this field? Is it “dangerous”?

  37. P. M. Claarke said on December 9, 2017 at 11:42 am
    Reply

    “He noticed that Firefox was not deleting ETAg data anymore when she cleared the cache”

    Something is not right with that whole sentence

    1. Tom Hawack said on December 9, 2017 at 5:36 pm
      Reply

      Looks as a smile to the fact that some users have wondered if Pants was a girl. So combining “he” and “she” is a way of entertaining the mystery :=)

      1. Pants said on December 10, 2017 at 5:55 am
        Reply

        Ssssh

    2. michlind said on December 9, 2017 at 12:22 pm
      Reply

      It. Shouldn’t she be “it”? ;-D

      1. b said on December 9, 2017 at 1:16 pm
        Reply

        I often wonder about the amount of women leaving comments/reading the blog. according to ads shown ( library pc’s dont offer adblocking ) on ghacks, the target is consumers biased by porn and pension scemes!

  38. d3x said on December 9, 2017 at 10:23 am
    Reply

    I have browser.cache.disk.enable set to false and was very surprised that this etag tracking is still working on browser restart, but after a little bit of research I found out it was probably because of Firefox session manager.

    Another thing:
    “A bug listing on Bugzilla@Mozilla that was created 14 years ago”
    This is a clear example of how Firefox developers care about fixing old “bugs”, I also saw this with Thunderbird where some small irritating bugs are not fixed for many many years. Great example was fixing 4GB limit with POP3 accounts or https://bugzilla.mozilla.org/show_bug.cgi?id=494100

  39. RossN said on December 9, 2017 at 9:47 am
    Reply

    Another option:
    Set your cache to a RAM Drive. I use the last free version of SoftPerfect 3.4.6

    1. Kane said on December 11, 2017 at 1:22 am
      Reply

      Addendum: For people who can’t easily change IP to test this, you can change your useragent instead.

      Clear cache = Delete ETags
      Change useragent and reload = Receive a different one. The server builds its ETag based on your IP and useragent, and possibly other headers such as accepted languages.

       

      I’m glad I finally got to checking this website and the undeletable ETag claim. I’ve had it far in the back of my mind for way too many years.

      1. Pants said on December 12, 2017 at 9:08 am
        Reply

        Still haven’t fire up TBB, but this is the only ticket trac I could find – https://trac.torproject.org/projects/tor/ticket/523 – its old, but I guess they at least mitigate the cache and other persistent data with each new id

        Also, this guy “bee” is nutbar crazypants .. maybe he took tool the red pill :) … https://trac.torproject.org/projects/tor/ticket/1579

      2. Pants said on December 12, 2017 at 8:58 am
        Reply

        I’d have to fire up my TBB (which I can’t do right now), but I think they disable disk & memory cache.

        As for Firefox, I asked Arthur to add said 14yr old ticket to the Tor Uplift (pasted here to save you time: https://bugzilla.mozilla.org/show_bug.cgi?id=231852 ). Some of the discussion in that ticket is enlightening.

        What I would like to see is something similar to canvas. When RFP=true, then automagically block etags via the response header, but promote an etag site permissions exception – same as they are doing with canvas see https://bugzilla.mozilla.org/show_bug.cgi?id=1413780

        And one day, under the new preferences UI section (where camera etc is now located), when RFP is true, also show Canvas, and Etags – so users can moderate their whitelists

        PS: are you the same “Kane” as this “cane” – https://anonymous-proxy-servers.net/blog/index.php?/authors/87-cane

      3. Kane said on December 11, 2017 at 10:21 pm
        Reply

        Regarding the need to block ETags on Firefox like you would with cookies, I am not sure.

        My hunch is that these days, the thing to do is block third party requests. If you don’t then ETags are a danger indeed, but I’m not sure what is more dangerous, during a given browsing session with a single IP, between:

        – Not blocking ETags and sending your IP and headers
        – Blocking ETags and sending your IP and headers, at the cost of installing one more add-on

        In both cases it seems that you’re screwed, the former might be uniquely identifying slightly more often but the latter has a level of unpredictability due to the need to rely on an add-on.

        In both cases the solution to get rid of tracking is to clear cache when you change IP. Blocking ETags doesn’t make one immune to other cache attacks so as long as there is a cache it should be cleared when IP is changed. But the best way is not to be tracked at all, so blocking third party resources.

         

        On certain configs I agree ETags are way more clearly a threat: It’s those where your IP changes frequently during a given browsing session without the cache getting cleared.

         

        > The best solution if for the Tor Uplift to apply something to privacy.resistFingerpinting so a sizeable number mitigate etags the same

        Indeed, that would be useful. Are they aware of it ? I doubt Tor Browser doesn’t have a patch that deals with ETags… And if they do, I don’t see why it wouldn’t be uplifted to Firefox. Have you heard anything ?

      4. Kane said on December 11, 2017 at 9:32 pm
        Reply

        > > THERE IS NO ETag PROBLEM.

        > Yes there is.

        You’re right, since ETags are effectively cookies that work even when cookies are disabled or deleted. The cache (or specifically ETags) needs to be disabled or deleted separately for this tracking avenue to be removed.

        That said, the cache even without ETags is already a tracking avenue that works independently from cookies, is it not ? Timing attacks, unique URL parameters and perhaps even uniquely crafted images come to mind, though I haven’t thought these out. It just has always been my understanding that the cache is a tracking avenue as much as cookies.

         

        What I had in mind when I said ETags were not a problem was that there is no such thing as undeletable ETags. If they really were impossible to remove through normal means that would have really annoyed me. IndexedDB I could take for many reasons, but ETags are way too pernicious…

        Also sorry for the caps even as the content of my post is poorly written. Just wanted to try and reassure people in a visible way, but maybe that sounded like some special, rather lame attitude instead.

      5. Pants said on December 11, 2017 at 7:45 am
        Reply

        Thanks for digging into the PoC. Using the IP (&UA?) to generate a unique etag is flawed, and the dev says some things about it all lower down in the page. As said in the github issue, it drove me nuts (that chicken and egg, grr) with some inconsistent results – as a result of this article and your (and other’s) digging, it’s a lot clearer, and explains the 2 count as well. We questioned the PoC almost 5 months ago.

        So the PoC is flawed, however, the concept is solid and well documented. A unique etag can be assigned and effectively read by the website on subsequent visits (until the cache is cleared, thank you for clearing that up), which is why this solution (albeit not perfect) works, because as you said it blocks the response header – which is EXACTLY what we aimed to do

        > THERE IS NO ETag PROBLEM.

        Yes there is. How wide spread? IDK. Disabling disk and memory cache is effective and is used for this purpose. Clearing cache is also used/recommended (see numerous older articles on this re etags). The threat model remains, and the solution above is one way to mitigate it, but not lose the full benefits of running cache. Does it make you more fingerprintable – that’s debatable – a unique etag WILL track you as it’s unique. So subsequent visits to a site without clearing cache will join the dots. Would I rather be unique, or in a subset of 2-3%?. Don’t get me wrong, this is but one piece in a large puzzle, and something that blocking unnecessary JS or third parties etc doesn’t have any sway over. Something that clears cache and changes IP every 10 minutes such as JoDonym is good. The best solution if for the Tor Uplift to apply something to privacy.resistFingerpinting so a sizeable number mitigate etags the same

        Anyway, getting off topic re FP’ing. Clear cache does clear the cache (thanks), but the etag tracking concept is a real threat, and this is a good solution depending on your needs. Others may want no cache, others may want to just clear cache every 15mins. All three work to differing degrees/side-effects.

        Anyway, thanks for the explanation

    2. Kane said on December 10, 2017 at 11:59 pm
      Reply

      So to sum it up, the only thing that needs to be cleared to remove ETags is the cache.

      Then upon reload, you will see the same text again even though the ETag has been cleared, because of the website “bug” described by the website:

       

      “The image is loaded after the page is loaded, but only the image contains the ETag. How can I display up to date info on the page? Turns out I can’t really do that without dynamically updating the page, which requires javascript, which I wanted to avoid to show that it can be done without.

      This chicken and egg problem introduces a few bugs:
      – All information you see was from your previous pageload. Press F5 to see updated data.”

       

      If you reload a second time without doing anything else, you will see that the text isn’t there any more. (But the ETag will have been saved again, of course)

       

      So to super sum it up:

      THERE IS NO ETag PROBLEM.

      Unless you consider it a problem that what is effectively a cookie is deleted with the “Cache” checkbox rather than the “Cookies” one. It is not a problem though, because even without ETags, the fact that you stored cache remains a tracking avenue in itself.

       

      As for the reason why the add-on listed in this article works, it’s simply because it changes the RESPONSE HEADERS, in other words the data received from the website. You don’t need to use this add-on if you clear cache with cookies. (It may also slow down your browsing by making the servers you connect to more likely to send resources you already cached)

    3. Kane said on December 10, 2017 at 11:37 pm
      Reply

      I’ll be very impolite and hijack the first comment, since everyone has apparently been confused

       

      ETags ARE CORRECTLY CLEARED WITH CACHE.

       

      What led to confusion could be the testing website: The ETag sent with tracker.jpg appears to be IP-dependent. So if you have a given IP and clear the ETag, you should get the exact same ETag sent by the server on reload, which means you will store it again. But you did delete it. Now If you clear the cache and then change your IP and THEN reload, you will get a different ETag.

      Note that in all cases, the ETag is cleared along with the cache and cannot be sent again to the server regardless of IP.

       

      Here’s a proper way to reproduce.

       

      – Open a new tab and press F12, and go to the Network tab

      – In this new tab, paste the URL http://lucb1e.com/rp/cookielesscookies/ and press enter

      – Watch how the click on tracker.jpg within the Network development tool, then click on “raw headers” in the right pane. You’ll see an ETag line in the “Response headers” field: It’s the value sent from the server to be retained by the browser. Copy paste it in a text file or whatever, just remember it.

      – Reload the page and check tracker.jpg again to confirm that the “Request headers” field has an “If-None-Match” line with the same value as the ETag you remember. This is the value sent from your browser to the server, the thing that can be tracked.

      – Now press CTRL+SHIFT+DEL so that the data cleaner window pops up.

      – In that small window, select “Everything” from the drop down menu and tick all check boxes.

      – Clear the data.

      – Check tracker.jpg. You’ll see that the request header “If-None-Match” has been cleared, but the response header coming from the server has the same ETag. It’s not something to worry about but let’s make it change to remove all doubts.

      – Clear all data again

      – This time, change your IP address

      – Reload the page and check the ETag from the response headers, notice that it has changed. If-None-Match is also absent from the request headers.

       

      In fact, you should see that checking only the “Cache” checkbox from the data cleaning window is enough to remove the ETag from the browser, as expected.

      Changing IP will make the server construct a different ETag for your browser to store, but as long as you have the same IP, the server will construct the same ETag. (Possibly it uses other headers too like useragent, it doesn’t matter) But the browser did clear the ETag along with the cache.

    4. Moloch said on December 9, 2017 at 3:06 pm
      Reply

      Indeed, you do have to manually close down the browser and start it again instead of using the restart option in for example waterfox itself so it clears the ramdisk.

      Using ImDisk myself.

Leave a Reply

Check the box to consent to your data being stored in line with the guidelines set out in our privacy policy

We love comments and welcome thoughtful and civilized discussion. Rudeness and personal attacks will not be tolerated. Please stay on-topic.
Please note that your comment may not appear immediately after you post it.