Former Microsoft Employee explains why bugs in Windows updates increased
Has the number of bugs in Windows updates increased in the past couple of years? If so, what is the reason for the increase in bugs? That's the question that former Microsoft Senior SDET Jerry Berg, better known as Barnacules, answered in a recent video.
Berg worked for 15 years at Microsoft and one of his roles was to design and develop tools and processes to automate testing for the Microsoft Windows operating system. He left the company after Windows 8.1 shipped to the public.
Microsoft changed testing processes significantly in the past couple of years. Berg describes how testing was done in the late 2014 early 2015 period and how Microsoft's testing processes changed since then.
Back in 2014/2015, Microsoft employed an entire team that was dedicated to testing the operating system, builds, updates, drivers, and other code. The team consisted of multiple groups that would run tests and discuss bugs and issues in daily meetings. Tests were conducted manually by the team and through automated testing, and if tests were passed, would give the okay to integrate the code into Windows.
The teams ran the tests on "real" hardware in a lab through automated testing. The machines had different hardware components, e.g. processors, hard drives, video and sound cards, and other components to cover a wide range of system configurations, and this meant that bugs that affected only certain hardware components or configurations were detected in the process.
Microsoft laid off almost the entire Windows Test team as it moved the focus from three different systems -- Windows, Windows Mobile and Xbox -- to a single system. The company moved most of the testing to virtual machines and this meant according to Berg that tests were no longer conducted on real and diverse hardware configurations for the most part.
Microsoft employees could self-host Windows which would mean that their machines would also be used for testing purposes. The main idea behind that was to get feedback from Microsoft employees when they encountered issues that they encountered during work days. Berg notes that self-hosting is not as widely used anymore as it was before.
The main sources of testing data, apart from the automated test systems that are in place, comes from Telemetry and Windows Insiders. Windows Insider builds are installed on millions of devices and Microsoft collects Telemetry from all of these devices.
If something crashes, Microsoft gets information about it. One of the issues associated with the collecting of Telemetry is that most bugs are not caught by it. If something does not work right, Microsoft may not be able to discern the relevant bits from Telemetry data. While it is in theory possible that users report issues, many don't and at other times, issues may go under because of other feedback that Microsoft gets from Insiders. Additionally, while Insiders may report bugs, it is often the case that necessary information is not supplied to Microsoft which poses huge issues for the engineers tasked with resolving these issues.
Tip: you can view the Telemetry data that Microsoft collects. Also useful: find out how to configure Telemetry on Windows 10.
Back in 2014/2015, Microsoft's Testing team would be tasked with analyzing bugs and issues, and supplying engineers with the data they required to resolve these. Nowadays, Berg notes, it is Telemetry that the engineers look at to figure out how to fix these issues and fixes are then pushed to customer devices running Insider Builds again to see if the issue got fixed or if it created new bugs.
One of the main reasons why Microsoft stopped pushing out new feature updates to everyone at once was that issues that were not detected by the processed could potentially affect a large number of customers.
To avoid total disasters like the Windows 10 version 1809 launch, gradual rollouts were introduced that would prevent feature updates from being delivered via Windows Update to the majority of machines in the early days of the release.
Microsoft exchanged the in-house Testing team with Telemetry data that it gathers from Insider Builds that it pushes to consumer and business devices, and replaced much of the PCs that it used for testing with virtual environments.
All of that led to an increased number of issues and bugs that customers face on production machines when installing Windows updates or feature updates.Â (via WinFuture)
Now You: What is your take on Microsoft's approach? What needs to change?
> Microsoft laid off almost the entire Windows Test team
and so began the downfall of Windows 10…Microsoft being stingy and pathetic as always.
The laying off of almost the entire Windows testing team is the main thing to takeaway from this article.
After Bill gates was no longer actively running Microsoft, I noticed a decrease in product quality.
Bigger troubles came with the current Microsoft CEO, Satya Nadella, who came to power about when the troubling changes described in this article were implemented.
Satya is the cause of ALL of Microsoft’s current problems and he is a traitor to his company. He is cannibalizing Microsoft one product at a time to appease his handlers at Google and Apple.
This explains why most of the bugs haven’t appeared on my Windows 10 installation… because it has been permanently quarantined to running in a VM! ;) That and I’m rarely in Windows anymore.
Having worked on literally thousands of machines, and too much hardware to even recall, throughout my life I know that each one has quirks and quite frankly a personality imho. Some are more agreeable than others… even to this day. Reminds me of people.
Sometimes automation just isn’t worth it and other times it can be. Finding the balance is usually the wiser thing to do “Grasshoppers”.
Windows 10 is OS designed for VM lol
Windows not a priority for Microsoft anymore. They decided that Insiders could do this testing for free and could eliminate internal paid testing. Has it worked out? No
Microsoft’s approach is completely antithetical to all good practices in software development: God knows why they did this, but there is literally no testing and everyone claiming that Windows 10 was a perpetual beta (or even alpha) appears to have been right all along.
Mozilla – take note.
Though I know they won’t.
There’s no big philosophy here – the basic issue comes from the typical corporate problem of “how to save precious muney” which lead to disbanding QA branch and moving all tests onto users – that’s their source of problems.
As long Microsoft with face of Nadella will pretend that issue of removing QA doesn’t exist and instead bringing it back testing at their place, will happily remain focused on only GUI changes (the yet another attempt of changing menu Start is the newest example) the issues will stockpile and lead to even serious problems in the future.
Microsoft became a pathetic greedy slug whose systems are buggy n glitchy and hog 40% of the gardware for stupid telemetry that NO ONE asked for… And still they take money for it… What needs to change: they should look at, why linux is SOOOO MUCH better and try to copy THAT i mean its infinitely stable, ist snappy and fast, it is safe and it doesnt make you rage or vomit… Microsoft is trash by this point
Microsoft is relying more on machine learning and big data (telemetry) to catch software bugs.
But you always get less quality when you replace a team of human experts with machine learning and big data. Many situations are complex and you need to understand the context to draw correct conclusions.
When dealing with complex situations, software algoritmes can only get as good as humans when they have the same intelligence as humans, .
The same problem applies to all fields where AI is used (language translation, etc).
Only in simple and specific situations are machine learning algoritmes better than humans.
The reason big tech companies are replacing humans with the current limited AI, is to reduce costs to increase profits. But the price paid is less quality for products and services. Society pays that price.
For these big tech companeis, greed is more important than the respect for customers.
The same thing is happening in computer security. Companies are trying to replace seasoned computer security experts and ethical hackers with machine learning.
It’s not going to work.
In fact, it’s been proven that hackers can subvert machine learning systems relatively easily. This is not surprising since “hacking” by definition involves thinking “outside the box” – whereas machine learning *is* a “box.”
People talk about “AI” as if was a real thing. It’s not. It’s merely a set of algorithms intended to do certain tasks which mimics certain forms of human processing. The problem is that human intelligence is much more complex. AI hasn’t come anywhere close to being able to process concepts like the human brain does – and without that, there is no such thing as “AI” except in a very narrow sense.
This is why all the “we’re in danger from AI” stuff is so much nonsense. Until computers can do conceptual processing the way human brains do it, humans will always be able to out-think “intelligent machines”.
That’s the whole point of the James Cameron Terminator movies.
Now, it *is* likely at some point that machine conceptual processing will be developed. At that point, the obvious answer to any “AI danger” is to import that capability into human brains rather than building real “intelligent machines.”
I’m almost literally the only person interested in the subject who has pointed this out. Everyone else is mired in “AI group-think.”
So what your saying is that they should be paying us to run windows on our PC’s.
The irony is that because most of the machines I buy have Windows pre-installed (which I immediately wipe, as I don’t use Windows on my own machines), for years I’ve been paying Microsoft NOT to use Windows.
Courts has long ruled that Microsoft should pay you ~$50 each time you wipe Windows from pre-installed Windows PC.
Yes, I know, and even got such a rebate once. The problem is that doing so takes a lot of work and patience — so much so that it’s not worth the $50.
What is the process for getting the refund?
@CKing123: I haven’t done it for years, so I don’t know what the current process is. Back when I did it last, I contacted the machine’s manufacturer (Dell, I think) and they had a process in place.
why, you do pay M$ for not using Windows? I order a computer “no O.S.” that saves me 60-80 $ in purchase price and I can install my favorite Ubuntu.
@ard: If I have a “no OS” option for the sort of machine I need, then I do that. But that’s not often an option.
@John Fenderson: “If I have a ‘no OS’ option for the sort of machine I need, then I do that. But thatâ€™s not often an option.”
From what I’ve seen, it’s *rarely* an option (and I’m sure OEMs get a better break on OEM licensing from Microsoft in exchange for that minor detail). And here’s what I think about *that*:
Tying arrangement | Wex Legal Dictionary / Encyclopedia | LII / Legal Information Institute
* * *
“Where a tying arrangement is unlawful, it may be illegal per se or illegal under the rule of reason. The requirements for a per se violation are: the forced purchase of one commodity in order to obtain a separate desired commodity or service; possession by the seller of sufficient economic power with respect to the tying product to restrain free trade in the market for the tied product; and that the arrangement affects a not insubstantial amount of commerce in the market for the tied product. If the requirements for a per se violation are not met, a tying arrangement may be illegal under the rule of reason if: it results in an unreasonable restraint on trade in the relevant market under Â§ 1 of the Sherman Act; or its probable effect is a substantial lessening of competition in the relevant market under Â§ 3 of the Clayton Act.”
Making a rebate economically unworthwhile to pursue is the same thing as no rebate at all, which amounts to outright tying. So I wonder: if the overwhelming majority of computers are sold conditioned on the de facto mandatory purchase of a Windows license, with Windows pre-installed, is it not reasonable to surmise that competition from Linux and BSD in the operating-system market might be substantially lessened? Clayton Â§ 3, beeatch!
(Don’t hold your breath waiting for the DOJ’s Antitrust Division and the Federal Trade Commission to see it that way. They’re too busy fending off threats from the congress-critters Microsoft owns, and weighing deferred payoffs from the various companies they’re charged with overseeing and regulating. From what I’ve seen, the EU Directorate-General for Competition isn’t yet so thoroughly captured. I can’t speak to European courts, but ours seem to have elevated Chicago-School dogma above black-letter antitrust law.)
That’s a great background and summary of the situation. Thank you!
” One of the issues associated with the collecting of Telemetry is that most bugs are not caught by it. If something does not work right, Microsoft may not be able to discern the relevant bits from Telemetry data. ”
1. If most bugs are not caught, why is it still being used? Its been in place since July 2015.
2. Seems that those who work for free have no value to offer. The solution to this is obvious.
Microsoft’s new approach will work great once everyone standardizes on the same h/w for every machine.
So to save a few dollars in internal teams they risk their reputation and the stability of production machines. Brilliant. Just brilliant.
It is pretty brilliant, actually. They know that they have enough of the market that they can get away with doing this and people will stick with their products regardless. They only have to be concerned about quality when there is a competitor breathing down their neck.
The problem is the telemetry requirement is a privacy risk as data has to be collected and sent to MS. This creates problems when privacy protection is legal requirement, can the user say with absolute surety that no privileged information got passed to MS unintentionally. Note, the law says the user is responsible for security not MS. So using Windows 10 in areas could be nasty lawsuit waiting to happen for some.
> Tom: “They know that they have enough of the market that they can get away with doing this and people will stick with their products regardless.”
Therein lies the danger of “immortal” monopolies that maintain their hegemony by all means. Instead of walking away, users would rather whine & remain held captive by Microsoft (or any dominant entity), because users fear that there is no viable alternative.
And that fear is not unjustified, since Linux-based OS is not (& perhaps will never be) user-friendly for the majority of people. Neither does it have sufficient hardware & software support.
Meanwhile, Windows-like OS such as ReactOS is still at alpha stage. And considering the glacial pace of development, there is no chance of it hitting RTM anytime in the far future.
One has to wonder: If they went back to the old way and got rid of the telemetry, instability, and out-of-control updating many people hate, how many additional people would upgrade from Win 7 and 8.1 to Win 10, not switch to other operating systems, and stop spreading ill will towards MS. This in turn would perhaps pay for the entire team and system they laid off. My guess is it would be overall worth it to MS to return to the old system.
With the internet, software became more buggy, because the developers could afford testing less and fixing bugs later with updates. With tracking (telemetry), software became even more buggy, because developers would rely on it to find bugs and test even less before releasing (Microsoft example here). Increasing profit at the expense of users.
That’s the answer to those who claim that tracking is necessary to develop better software. Tracking does not only violate privacy: it effectively makes software more buggy.
Very, very well-said!
That’s a great observation!
Microsoft clearly needs to bring that testing team back. That telemetry data is insufficient to detect and fix bugs in not surprising — the surprising thing is that Microsoft (and a distressingly increasing number of other companies) appear to have honestly thought it was.
The reality, of course, is that telemetry-based systems are preferred by companies because it’s cheaper, not because it’s better, but apparently Microsoft (and many others) have fallen into the trap of believing their own PR.
This is from MS’s 2018 Annual Report…
Our ambition for Windows 10 is to broaden our economic opportunity through three key levers: an original equipment manufacturer (â€œOEMâ€) ecosystem that creates exciting new hardware designs for Windows 10; our commitment to our first-party premium device portfolio; and monetization opportunities such as gaming, services, subscriptions, and search advertising.
Microsoft’s ambition is to broaden its Win10 economic opportunities, not make Win10 a better operating system. MS seems to believe the aphorism that “perfect is enemy of good.” Expect MS to keep foisting bugs onto your computers. I don’t think they care anymore.
This is not a spoiler, as we knew for ages the changes microsoft made with testing, well assumed and happy with.
Microsoft now track their visitors on their websites with HTML5 fingerprinting ! Are You still a Microsoft fan ?
@stefann: “Are you still a Microsoft fan?”
What do you mean, “still”? ;-)
Except OEM’s preload Windows with a lot of garbage especially their own stuff and then they stop updating their own products within a year or two and drivers once the product is no longer offered.
Everything that comes through our doors that is OEM preloaded is wiped and freshly installed.
A viewpoint of a former QA at MS…
This part of that post really caught my eye:
“The Dev’s never included the test team (my team and sister teams) in any email threads or code reviews or hallway discussions regarding anything. Usually we figured out some feature of the product changed when our tests started to break. We never knew what was going on because no one every bothered to invite us in or include us in any conversation.”
That utterly blows me away (and goes a long way to explaining why Microsoft’s quality control has never been great even when they could be bothered to do testing).
I’ve been a software engineer in this industry for a very long time (~30 years), and one of the things that I’ve learned is that SDETs are properly part of the development team and need to be treated as full equals there. To do less is to pretty much guarantee problems in the product.
In fact, when I’m interviewing for a position, this is one of the things I ask about because it’s a HUGE red flag if testing is considered or treated separately from development.
I’m never being Microsoft’s lab rat. Windows 10 Enterprise LTSC is the way to go, or stick with Windows 7 and update with 0patch. Its time to start playing with alternative operating systems. Windows will be a disaster long as Satya is in charge.
You don’t need a virus to spoil a windows PC. Microsoft create their own viruses…but they call it “updates”
In other words, MS are a bunch of cheap bastards. Good to know.
Here I am using Windows 10 with nary a hitch. Guess I’m in the minority.
Nope. Same here. I’m using three (3) Win 10 Pro x64 machines with telemetry turned off and “quality” and “feature” updates delayed for weeks. Never had a problem.
I clearly see Satya Nadella as a far worse CEO than even Steve Ballmer as far as products like Microsoft Windows and Office are concerned. Only because Azure brings in enormous revenue – it is a new multibillion dollar business for them in addition to Office 365 (I can’t imagine who are the retards paying for Office as a service), the company’s stock price has skyrocketed. That’s why they have no incentive left to improve Windows any more. They were always a customer abusing monopoly, but now they have several additional enterprise-only businesses, so end user computing or “more personal computing” as they call it gets screwed even more. Unless some OS comes along that is an actual threat to Windows, nothing is going to change.
This article and this whole thread just floors me. How soon they forget…
One of the things that drove me to Linux in the first place was the interminable incompetence of the late-Gates era and entirety of the Ballmer era. Among the dubious “accomplishments” I count:
– Millennium Edition – most likely the worst operating system I have ever used
– The Longhorn debacle – where proposed features were regularly subtracted over a protracted period, eventually leading to the buggy initial release of Vista
– Windows CE/Mobile – inferior to Palm OS, iOS, and even early Android (and, yes, I had to actually support this garbage)
– Zune – no need to say more
– Windows 8 – despite shoring up the underlying operating system, Microsoft torpedoed this product with a confusing, nearly unusable OS (while mostly ignoring what pre-release testers had told them about usability)
– Windows Phone – the incompetence of Windows 8 shrunk to mobile size… and you thought Windows Mobile was bad!
– Windows 8.1 – “An interface that sucks slightly less than Windows 8! You can actually use it!”
– Windows 8.1 u1 – “We can’t really make this awful UI suck much less, but we tried! Here is an idiotic naming/numbering convention to help you keep track!”
… not to mention numerous Windows Updates that dropped KB “fixes” that hosed something important and were a major pain to uninstall/reinstall or where the OS needed to be told to ignore or even required reinstallation
…not to mention raging mediocrities like Windows 2000, Windows XP, and Windows 7 (arguably the most overrated Microsoft product ever).
Take a trip down Memory Lane, with me, and visit the “glory” days when Ballmer, Barnacules, Windows 7 and their ilk ruled Redmond (please note the popularity of articles advising on how to fix things broken by Windows Update)…
That guy is so annoying it’s unwatchable. Blabber, blabber, should go to the barn and stay there.
Perhaps my feelings about Microsoft’s new approach to testing and patching are best illustrated by a story. I had just voiced my misgivings about the direction Windows was taking at a public shareholders meeting, and Satya Nadella came down off the stage with a young relative to confront me. It went like this:
[Nadella:] Your reputation is well deserved, captive user. I don’t think there’s ever been a Windows critic to match you. As for this young man, he insists you are Richard Stallman reborn. Or was it Simon Davies? Why doesn’t the hero reveal himself and tell us all your real name? You do have a name.
[Me:] My name is User. [I turn away from Nadella and go back to my laptop screen.]
[Nadella:] How dare you show your back to me! Captive, you *will* remove your hat and tell me your name.
[Me, after taking off my hat and slowly turning around to face Nadella:] My name is Maximus Decimus Meridius, Commander of the Armies of the FOSS, General of the Linux Legions, and loyal advocate for the *true* Emperor, the End User. Former user of a murdered OS, former owner of a murdered laptop. And I will have my vengeance, with this distro or the next.
What? You didn’t like my story? You thought it was lame? Were you not entertained? WERE YOU NOT ENTERTAINED? ;-)
(C’mon, guys. Cut me some slack. It sounded a *lot* cooler in my head, with Joaquin Phoenix and Russell Crowe doing the voices.)
I think their exchange also is an engagement for Windows Insider’s users. They can have ability to get new features like we’re having with web browsers, and they provide the telemetry as a big data of polymorphism of hardwares in the world for Microsoft. Fair enough. I’m feel happy with Windows 10 more than Windows 7 in the stability, even my PC already 10 years old.
Did anyone catch that news item, about a microshit employee, playing the part of an unofficial market researcher, who is attempting to determine why people hate microshaft so much ? LMFAO !
This is a bit like, crossing the street with the intention of kicking shit out of someone for no legitimate reason, and five minutes later asking the victim “are you alright ?”
This is the sort of psychotic mentality that Windows users have to deal with every day, and the cash hungry lunatics in the Linux community, whom are allowing themselves to be progressively seduced by microshit dollars.
Lets put it this way, would you invite a psychotic, parasitic, vampire into your home, after it had murdered most of your family, and then apologized to you for the distress it caused ?
Peter Newton [London UK]
You are Microsoft’s free beta tester.
Currently working as a contract worker at MS.
Have been one for the past decade off/on due to their hiring policies and changes in regards to vendors. (Currently it’s work 1.5 years and take .5 off, but your position often gets filled in the .5 off so you end up in different teams since each time is finding a new job/contract)
Anyways, I started as a tester. There was lots of TCRs and other cert requirements to follow. The products had most of the bugs fixed. Then over the years they started requiring less and less. Then they started laying off more and more testers.
Now I am about to enter a new .5 year long break and the future is looking worse and worse for MS. They won’t fix the issues. Even if someone on the inside reports a 100% repro bad issue.. it won’t get fixed most of the time. More and more teams are seeing budget cuts, and not just the ones you would expect. (E.g. receptionists, cafeteria, shuttles/busses)
I see the whole thing slowly deflating like a bubble. FTEs go off for 30+ vacation days, managers not knowing what their underlings are doing, and yes.. testing is all but a dream here anymore. It does upset me since there is nothing I can do despite being on the inside.
In all honesty I am going to start learning more and more Linux, cause I doubt windows will be the top OS in use in 20 years.. if not 10.
Hopefully MS will learn from their mistake and start focusing more on their core OS and start fixing bugs.. but learning is not something my manager.. or her manager, or even that person’s manager does well. They want it done the way they say.. even if it’s wrong or you know a better way. Wasted 4x hours doing something my manager didn’t even need done because they didn’t want to spend 5 minutes finding the email with the correct information.
Things change.. don’t trust your life to MS. They are dropping the ball and have no plans to pick it back up.
What is your recourse if the update breaks your computer, damages your registry and makes your computer unable to play sound? And once your ticket is open and your issue is known, subsequent efforts to contact support are ignored?