Mozilla creates Shield study rules to avoid another Mr.Robot disaster
Mozilla created a set of guiding principles for Shield studies after it launched an analysis of the Looking Glass Shield study which went wrong on several levels.
Looking Glass was released as a system add-on to Firefox which meant that users saw the add-on appear in the browser's add-on manager without them initiating the installation.
This appearance out of thin air was arguably the biggest complaint that users had and something that got them alarmed because it shared the characteristics of malware. The fact that the initial description and add-on name did not reveal anything about the add-on's purpose added to the confusion as well.
The description of the add-on read "MY REALITY IS JUST DIFFERENT THAN YOURS" and "PUG Experience Group" was listed as the creator. Nothing linked the add-on to Mozilla in Firefox's add-on manager.
Mozilla announced shortly after the study blew up in the company's face that it had pulled the study, uploaded the add-on to Mozilla's Add-on repository, and started an investigation to "better understand how and why this happened", and how the company "could do better" in the future.
Shield studies are used to A-B test things in the Firefox web browser. The studies are used to test small and big changes to Firefox, from simple icon or color changes to new features, to find out whether the majority of users that have been selected for the study like the changes, or not.
Studies help Mozilla make better decisions in regards to future changes and features in the Firefox browser.
This platform helps us make decisions on new product features, evaluate whether or not a technology update is stable, and generally helps us make sure that we can make good decisions in a responsible way.
Looking Glass did pass the privacy review as it did not collect any data. The fact that it did not should have been a red flag, as it is impossible to evaluate a feature without collecting a bare minimum of data.
Mozilla create a set of principles for Shield studies so that something like Looking Glass won't happen again.
- All Shield studies must answer specific questions.
- Shield studies will always respect user privacy.
- All Shield studies adhere to the "scientific method for answering complex questions".
- All Shield studies require a Product Hypothesis Doc which outlines the research question the study is trying to answer.
- All Shield studies must be named accurately.
Looking Glass would fail in all but the second.
Closing Words
It is definitely a good thing that Mozilla created a set of guiding principles for Shield Studies. I would have preferred if the organization revealed a bit more about Looking Glass itself: was Mozilla paid for the promotion and how did not anyone object to the study by pointing out the, rather obvious, issues it had?
Related articles
- Firefox 58: Mozilla will collect only base Telemetry data (release channel)
- Mozilla plans to collect anonymous Firefox browsing data
- What is Firefox Pioneer?
They are unethical. I deleted Mozilla Firefox from my system when they deleted Brendan Eich from their payroll.
won’t install; can’t trust them.
It’s pretty easy to check that such a small add-on doesn’t leak data. If you don’t trust them you shouldn’t use Firefox or any of its forks, and probably you shouldn’t use Chromium and any of its forks either.
They had a fairly strict policy and set of rules before. Those rule were apparently ignored. Does making a new set of rules decrease the odds that they’ll be ignored again?
Same as when a recurrent type of security vulnerability is discovered: It prompts involved people into adopting better development practises leading to less mistakes, and if it’s bad enough it prompts them into implementing an architecture that doesn’t just rely on people following guidelines.
I hope you’re right. The Mr. Robot thing didn’t exactly look like a mistake, but more of an intentional subversion of the shield study program.
I think this Looking Glass thing was not a study, but a game that just happened to be distributed through the Shield Study mechanism, which was not made for that purpose. It was advertising Firefox to Mr. Robot fans, not the other way around, as the add-on was completely inactive unless the user did something particular as prompted by one of Mr. Robot’s episodes.
As such, if anything it’s Mozilla who should have paid Mr. Robot’s I.P. owner rather than the other way around.
@People: Untick all checkboxes at about:preferences#privacy-reports and you’re opted out of these.
To protect from bugs where the main prefs would fail to be respected, you could additionally set these prefs in about:config.
Firefox 58.0.1 : I do have the last two settings in about:config (already emptied/disabled) but not app.shield.optoutstudies.enabled : maybe a hidden one…
As shown by Pants for the app.shield one, these are second line prefs, meaning that if the main pref(s) they are under is disabled, it doesn’t matter what they’re set to.
It’s not always a given that they should be disabled but in that case, since a lot of work is currently ongoing regarding how telemetry is organized, I’d rather be protected from bugs where the main pref is erroneously bypassed. If second line (more granular) prefs are disabled as well such a bug doesn’t matter :)
Tom its controlled by other prefs – https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/docs/internals/preferences.rst see lines 37 and 41
Lines 37/41 acknowledged, thanks Pants :=)
I think this Looking Glass thing was not a study, but a game. It was advertising Firefox to Mr. Robot fans, not the other way around, as the add-on was completely inactive unless the user did something particular as prompted by one of Mr. Robot’s episodes.
As such, if anything it’s Mozilla who paid Mr. Robot’s I.P. owner rather than the other way around.
What? Advertising Firefox in Firefox? No, it’s not the first time Mozilla tried to “diversify their income sources”. Think of the first Pocket integration or Hello sponsored by O2.
No system add-ons here, the [Mozilla Firefox install folder]\browser\features folder is emptied on every Firefox install, systematically. No experience, no study, I’m a browser user, not a tester.
I have prior experience writing helpers which use the Linux inotify interface to act on things like “write-mode file handle closed” or “temporary file renamed to its final name”, so I’ll probably automate that.
Some nice little tool to filter out the ones I don’t need and don’t want as the updater pulls them in. It’ll make a nice complement to pinning the about:config keys and translating my Classic Theme Restorer tweaks into userChrome.css rules.
Some of the features in that folder are useful though, such as the Web compat one which fixes website incompatibilities.
People may enjoy Screenshots and Form autofill, or maybe Activity Stream. I disable the last two in about:config. The only system add-on I delete is Follow on search since I haven’t verified in the source code that it respects the main telemetry switch and I’m not aware of an about:config pref that disables it either.
You’re a very naughty browser user, Mozilla doesn’t like that. I can hear them frowning in disapproval. Don’t tell them what features you want added to Firefox, because they’ll mark them as WONTFIX just to spite you.
I humbly beg their pardon as I finish coffee and sandwich.
Users’ reactions concerning system add-ons must make some devs frown more than once :=)