Microsoft's data-focused approach to Windows Updates requires an update
Microsoft uses a data-focused approach to determine if updates are ready for wider distribution and it appears that the company is very content with the approach.
Issues of recent updates suggest that Microsoft may need to rethink its strategy when it comes to Windows Updates.
Microsoft started a new series on the official Windows Experience blog some time ago in which company representatives lift the veil in regards to the company's strive for quality.
Microsoft told the public that update quality is better than ever, and that the same is true for the quality of released device drivers.
In "Data, insights and listening to improve the customer experience", information is provided on how Microsoft uses data to determine the quality of updates.
The fundamental question that Microsoft asks for each release is "Is this Windows Update ready for customers?". Updates go through different stages during development:
- Automatic and manual testing of updates.
- Evaluation of quality based on diagnostic data and feedback from Microsoft engineers.
- Distribution to Windows Insiders and further evaluation of updates.
- Distribution to the general Windows population.
The metrics that Microsoft gathers and monitors need to be equal to or better than the metrics of the previous update.
By the time we are ready to ship to our customer base, our metrics must be, at a minimum, at or above the quality levels for the previous release, the idea being that every update should make the Windows 10 experience better.
Is the approach sufficient?
Data plays an important role when it comes to the release of updates. Microsoft, and any other company for that matter, may use it to make sure that certain features behave as intended. It is certainly possible to catch bugs by just looking at data but if you look at recent updates, you will notice that things were not as smooth as Microsoft's data focused appraoch suggested.
Microsoft had to pause the Windows 10 version 1809 feature update because it caused a lot of issues. To name just a few:
- A Data loss issue
- Another data loss issue
- No audio issue and bluescreens
- Font display issues
- Mapped network drives issues
All of these issues were not detected during tests conducted by Microsoft, by diagnostic data, and by feedback that Microsoft engineers and participants of Microsoft's Insider program provided.
A game performance related bug was not detected in the most recent update for Windows 10 version 1809. Microsoft had to update the support article to add the performance affecting bug to the list of known issues of the update.
The problem
Microsoft engineers and participants of the Insider program may not provide a sufficient sample size to provide data for all major use cases. Gaming might be such a case. It seems unlikely that Microsoft engineers spend a lot of time playing games on their devices. Even if they would, they could never test new versions of Windows on tens of thousands of games that are available for PC. The bulk of Insider participants may not be interested in games as well. Gaming is just one area where Microsoft's approach falls short.
It is certainly unrealistic to expect Microsoft to catch all issues in all updates before release. The sheer number of hardware and software configurations makes that an impossible task.
But major issues, like game performance in popular games, should not hit the general population.
That's one reason why it is a good idea to install updates a while after release and not as early as possible: you never know what is going to happen.
Now You: Let me know what you think about all of this, and how Microsoft could improve update quality.
If Stage1=fail
goto Stage4
Nice one.
You joke but it’s true :)
[[[ 3. Distribution to Windows Insiders and further evaluation of updates.
4. Distribution to the general Windows population. ]]]
.
.
M$ force auto-collects Telemetry & Data from all Win 10 computers.
After nearly 4 years, the number of wide-eyed Windows 10 Insider alpha-testers should have dwindled to only a “handful” by now, ie compared to the frenzy of 4 years ago. It may no longer be viable for M$ to rely on Windows Insiders as alpha-testers for the evaluation of updates.
Win 10 Home computer-users are unpaid beta-testers who get served with M$’s forced auto-updates/upgrades as soon as they are released. Win 10 Home computers are usually low to mid-end computers.
……. Win 10 Pro & Ent computers are usually high-end computers, eg used by wealthy businesses and gamers. Most Win 10 Pro & Ent users defer forced auto-updates/upgrades by many days or many months. So, there will be scant Telemetry & Data collected by M$ from high-end or gaming Win 10 Pro & Ent computers when the forced auto-updates/upgrades are released.
If a forced auto-update/upgrade Blue-screens or borks a Win 10 Home computer to become unusable or inoperable or non-functioning, how will M$ still be able to collect and collate such data about the buggy update/upgrade.?
M$ seems to have overlooked certain factors about Point 3. & 4 and the changing time/trend, resulting in buggy updates/upgrades in Win 10.
……. To have actual quality updates/upgrades, M$ may need to reset or re-form the Windows Testing Division, at least for high-end Win 10 Pro & Ent computers. In 2014, M$ used to have 150 professional testers and 1200 lab computers to test Windows updates before release.
It would be even better if M$ resets Windows to the Win 7/8.1 model, ie no forced auto-updates/upgrades = fewer computers borked by her buggy and low-quality updates.
It looks to me like Windows QA process is resulting in substantially lower quality releases than they produced in years prior. They may want to consider at least going back to that.
I think this is a more extreme example of a problem that has become common across the industry, though. Part of it is the “rapid release” fad, which leaves precious little time for effective QA. The widespread adoption of Agile development processes, which is what spurred the rapid release trend, is one of the most significant problems here.
There’s also the problem that comes with telemetry. I’m not talking about the privacy aspects around that, but the misuse of the data. It is quite common that telemetry data misleads development in two key ways:
First, it encourages mindset that the only things that are really important are those things that can be measured with telemetry when, in fact, there are numerous incredibly important things that cannot be so measured. Related to that is that it encourages developmental laziness of relying on telemetry rather than thoughtful design, testing, and observation.
Second, it encourages “development by numbers”. This is not so much a quality issue in terms of reliability, but a quality issue in terms of functionality. The problem being when developers start to think that a capability isn’t important because it isn’t used very often. That’s simply logically unsustainable. As an example of why, the fact that fire extinguishers aren’t used very often does not imply that nobody should bother having fire extinguishers.
I could go on and on with this topic, but I fear I’ve already written a small essay, so I’ll stop here.
I will stay with Windows 8.1 until it is retired, and then rely on Ubuntu or some other Linux distro.
Why be a masochist? Microsoft will not change their policies, as long as people blindly follow them. There are some non-Microsoft choices (Mac, Chromebook).
I have a Ubuntu computer now.
Microsoft : Data, insights and listening to improve the customer experience
https://blogs.windows.com/windowsexperience/2019/03/06/data-insights-and-listening-to-improve-the-customer-experience/#9rjQE3BEGs8drguM.97
Once I worked for someone who came from the consumer software industry and his take on product introductions was “Get it out there, the customer will figure out if it works or not.” We made medical products in a regulated industry and fortunately his approach was not even possible.
I keep reading about how MS claims bad windows updates are only experienced by very small percentages of the user base. Meanwhile, site after site, credible sites, detail the latest update massacre with advice on what to do.
Decades later, consumer software is still an immature industry, the internet is worse and for some bizarre reason I still don’t understand, advertising is funding a huge amount of this nonsense with ads no one reads. Seems money flows among advertisers instead of up and down. Not sustainable but who cares if you can cash in big quickly.
How to make windows updates stop scaring users? Make them work right, it’s not hard and only do updates that fix problems or improve existing features. Releasing updates on a ridiculously frequent schedule just because guarantees regular fiascos.
How to get MS to do those things? Not possible with their current bent, their product development days are long gone, now they just sustain an OS that hasn’t fundamentally changed, just become more capable over time by adding crap to it that explodes.
While focusing on selling data from analytics that users mess with as much as possible.
Nice that number analysis strategy from Microsoft it sounds more and more to a user being a tester.
But I also have a question does anybody knows when the analytic number are safe enough that Microsoft is considering to upgrade main Version 1803 build 17134.590, to an 1809 version or maybe even directly (?) to the new release who is upcoming next month.
I am asking this because I have read not only here at Ghacks.net there will be a good chance that with the new 1904 update the Specture protection going is to be handled much better than with the protection against it right now?
Does anybody know?
Review the askwoody.com website for advice on latest updates, and consider using a third party program like pf100 Wrapper Script to control updates to your schedule. The Wrapper Script has option to use WUMT or WuMgr. I am only suggesting to ‘Consider’, because many have objections to third party update blockers. I find it useful for ‘Controlling’ updates, others have different opinions. Discussion is good. Some info at: https://www.majorgeeks.com/files/details/wumt_wrapper_script.html
Data-focused. Metrics. Telemetry. LOL They could probably come up with a way to make slavery sound appealing. Something like Consumer-focused Productivity Management.
Microsoft should have done day one with Windows 10, release the upgrade every two years, just to get the OEMs to catch up. Continue releasing updates and security fixes. Plug those back doors. Limit the bloatware.
Bricking a user’s computer is not only aggravating, it’s a frickin nightmare. Every windows computer is optimized to RAM, OS and drivers installed at the factory settings. Upgrading the OS can lead several hiccups. Back in a day when I upgraded from windows 8 to windows 8.1. A dude over phone gave me instructions to follow, I had to update the blu tooth driver via modifying the driver to get it restart properly.
Microsoft is being thick headed in getting more users to their OSes and pissing the hell out of them. Most people can barely afford a 400.00 laptop or sometimes rely on hand me down machines. 500GB HHD space is good enough for me. I am considering of switching to Linux once the windows 8.1 expires.
It feels good not jump the Microsoft’s crashed Windows train any more. I still use Windows XP (updated until April 2014), XP x64 (updated until April 2014) and 7 x64 (updated until May 2017) and will stay with them as long as my computers work. No, i don’t see the rush to DOWNGRADE to anything later than them. You can always lock down Your older operatingsystems with f.ex Faronic’s Deepfreeze or secure it other ways. I always laugh when i read how much trouble many people have with f.ex Windows 10. Even i would choose Linux before Windows 10.
Here some facts about Faronic’s Deepfreeze Standard : https://filehippo.com/download_deep-freeze-standard
Martin, maybe Deepfreeze is something You should write about !!!!