Microsoft says it’s improving patch quality for a complex ecosystem

Topic

New Reply

Michael Fortin just posted this on the Windows blog: While our measurements of quality show improving trends on aggregate for each successive Windows
[See the full post at: Microsoft says it’s improving patch quality for a complex ecosystem]

7 users thanked author for this post.

Elly, lurks about, Noel Carboni, PhotM, geekdom, GreatAndPowerfulTech, Mr. Natural

Reply | Quote

Replies

People need to read the 1954 book “How to Lie with Statistics” to recognize that metrics can be selected to make just about anything look better than it actually is.

GreatAndPowerfulTech

8 users thanked author for this post.

Elly, lurks about, woody, fk5353, SueW, Cybertooth, HiFlyer, samak

Reply | Quote

This book is still a classic and MS’s technique with this graph is straight from Chapter 5.

Doesn’t the graph look good, the incidents appear to be almost at zero currently, right? Wrong! The incidents are around 0.5k but since the graph starts at 0.4k it only has a short column representing 0.1k. If it was drawn honestly the columns on the right would be 5 times the size they are.

More MS lies and BS.

Windows 10 Home 22H2, Acer Aspire TC-1660 desktop + LibreOffice, non-techie

8 users thanked author for this post.

johnf, AlexEiffel, Elly, woody, SueW, GreatAndPowerfulTech, Chronocidal Guy, HiFlyer

Reply | Quote

samak wrote:

This book is still a classic and MS’s technique with this graph is straight from Chapter 5. Doesn’t the graph look good, the incidents appear to be almost at zero currently, right? Wrong! The incidents are around 0.5k but since the graph starts at 0.4k it only has a short column representing 0.1k. If it was drawn honestly the columns on the right would be 5 times the size they are. More MS lies and BS.

I went on a bit of a statistical excursion using that chart.  Note, I have no actual data other than that chart graphic, but it’s fairly easy to estimate the numbers used, so I reconstructed the chart to play some number games.  DISCLAIMER: I have no actual data for any of this, just a visual estimate of the original chart, and other published statistics.

Here’s an estimate of original chart, rescaled to use 0 as the baseline.

Not all that different really, it just shows a slowly decreasing trend.

But wait.  What units are they even using here?  “Hundreds per millions of devices” is about as clear as mud.  I suspect that they actually meant “per hundreds of millions of devices,” but just on the off chance they’re trying to obfuscate the issue that way on purpose, I’m going to run with that worst case.

“Hundreds per millions of devices” means we need to multiply that entire chart by 100 to get the number of reports per million devices.

Not looking so rosy now.  Those are really big numbers.  But in terms of percentages, I feel like the numbers might not be that far off, due to the trends at release.  Would a user complaint rate of 10-15% seem reasonable during the GWX nonsense?

Now, consider all the announcements made concerning how many millions of devices Windows 10 is running.  To get a running total over the whole life of Windows 10, I took the dates of the assorted “Windows 10 now on X million devices!” headlines, and linearly interpolated between them, running backward to an initial estimate of 100 million at release, because it fit the overall trend.  (If anyone has hard numbers for this data, please correct me as you see fit!)

In all fairness, it does seem pretty reasonable to normalize the data by number of devices, due to the ever-increasing rollout pace.  But from a device-centric perspective, what’s the grand total of incidents reported?

No more decreasing trendline.  That total is approaching 5% of all Windows 10 devices.  Honestly though, that doesn’t sound all that remarkable from a product with this wide a userbase.  It’s still a huge number, but 5% sounds like it would be comparable to the defect rate people might expect in a product.  Of course, this doesn’t account for things like severity of the error reported, or repeat occurrences, or whether the issue was ever resolved.  But it’s interesting to guess at what the numbers behind Microsoft’s statistical misdirection really are.

Note, I would disregard that first six months entirely, since it’s based on very flimsy estimates of Windows 10’s initial rollout.

If anyone has corrections, observations, or complaints, I’d love to hear them.  The language behind the “hundreds per millions of devices” issue in particular could potentially remove a lot of zeroes from those totals, so I wouldn’t call this breakdown more than a WAG at best.

 

10 users thanked author for this post.

HiFlyer, BobbyB, samak, AlexEiffel, Elly, lurks about, Cybertooth, woody, fk5353, DAVe3283

Reply | Quote

While I applaud your efforts to reinterpret the graph, it doesn’t  require that level of commitment to know that MS is a Master of Obfuscation.

Reply | Quote

I read many years ago that up to five percent failure rate for a electronic product was (maybe still is) an acceptable incurred loss. Is that their measurement of success?

Reply | Quote

Thanks for the effort and proving the blather was marketing ahem. I would say for an OS a reported error rate of 5% is way too high; that is 1 in 20 installs. Also, I would expect the errors to be underreported, even the most serious ones. So the real rate is probably well north of 5% and is anyone’s guess. Your analysis shows that whatever MS is using for QA it is a dismal failure.

1 user thanked author for this post.

HiFlyer

Reply | Quote

Severity is what matters.  There’s many times where an OS with an error (or defect) can still run normally in most circumstances, while hardware would just fail.  Due to the complexity, software in general will have more defects but they can usually run with them and remove them at a later date.  Defect rate is higher than 5%

Once something manufactured is shipped, it’s shipped.  It can’t be changed and will generally wear out given time.  5% “failure” rate is just that, failures not defects.  Manufactured goods generally will have defects in them that do not cause failure during their intended lifespan, however they may be what causes the device to fail after its guaranteed or expected lifespan.

Reply | Quote

You know the term “work-around?” Microsoft has invented something new: the “talk-around.” This announcement is just another, obfuscating, over-complicating talk-around, making it seem as though the simple and obvious solution–rehire and adequately fund a well-resourced quality-control/beta-test department–wasn’t right there in front of them all the time.

4 users thanked author for this post.

samak, mindwarp, Noel Carboni, HiFlyer

Reply | Quote

How to use up a company’s reputation, built over a lifetime: Say things, instead of doing things. It’s cheaper! People will take a Long Time to let go of belief.

-Noel

3 users thanked author for this post.

HiFlyer, samak, MikeFromMarkham

Reply | Quote

“We shifted the responsibility for base functional testing to our development teams in order to attempt to deliver higher quality code from the start, but often fell short as exemplified by the numerous issues both business and end users have experienced including, but not limited to, data loss.” 
There, fixed THAT sentence.

2 users thanked author for this post.

samak, HiFlyer

Reply | Quote

The very first sentence tries to minimize a huge problem among a few other serious problems noted with the 1809 release. My eyes started glassing over as soon as I read that.

In paragraph 3 he mentioned all the variables Microsoft has to deal with. It certainly is a very difficult situation to manage and deal with. It’s true that there’s bound to be some systems affected negatively by an update when dealing with that kind of volume. With all those variables to deal with maybe Microsoft should focus more on the issues at hand instead of this constant push for new features that most people have no interest in.

Looking at the “quality improvement” chart doesn’t really tell the history of how patch quality has steadily declined over time leading to much more serious issues resulting from the more recent updates.

We already know Microsoft has ignored warnings from whatever testers they have now. I would like to know how testing now differs from how it was managed before WaaS. I’ll be interested to read the promised forthcoming post giving more details on the current “testing process”.

 

Red Ruffnsore

2 users thanked author for this post.

HiFlyer, gkarasik

Reply | Quote

Unbelievable… this chart is misleading to the extreme.

For accurate figures, note how the vertical scale stops at “0.4k”. So in your head add about the vertical size of Dec ’15 in the chart to each of the bars. That paints a different picture, no? It would be nice if someone created the alternative chart based at zero to paint the true picture.

Let us also assume that “0.4k” means “400 per million devices” and not “400 hundreds (40000) per million devices”, which would be correct under a strict interpretation of the chart’s scale description. I refuse to believe they would be that misleading…

Also note that in absolute numbers, we’re still talking ~400 per million devices for 700 million W10 devices, i.e. 280k incident reports in last October alone!

And the customer satisfaction alluded to in the article can equally be explained by many people who are so disappointed that they seek ways to avoid updates altogether, instead of providing feedback on their marvellous experience. “You recently scored us 0/10. Please take a moment to give us your feedback again.” Uh-huh…

2 users thanked author for this post.

samak, HiFlyer

Reply | Quote

This is nothing but a crock and a bunch of excuses for MS nerds that have lost touch with their customer base – especially the corporate customer. I am sick of Microsoft making excuses rather than listening and making the changes necessary for stability. I don’t give a d*** how pretty a window looks – just that it consistently works even after patching

2 users thanked author for this post.

Noel Carboni, HiFlyer

Reply | Quote

quality
complex
ecosystem
validate
trends
approach
critical
attention
care
mission
empower
scale
dynamic
key
achieve
spectrum
goal
indicator
stabilization
evaluate

Were any buzzwords missed?

On permanent hiatus {with backup and coffee}
offline▸ Win10Pro 2004.19041.572 x64 i3-3220 RAM8GB HDD Firefox83.0b3 WindowsDefender
offline▸ Acer TravelMate P215-52 RAM8GB Win11Pro 22H2.22621.1265 x64 i5-10210U SSD Firefox106.0 MicrosoftDefender
online▸ Win11Pro 22H2.22621.1992 x64 i5-9400 RAM16GB HDD Firefox116.0b3 MicrosoftDefender

3 users thanked author for this post.

MikeFromMarkham, HiFlyer, BobbyB

Reply | Quote

They missed “truth”, “accuracy” and “honesty” to name at least three… but then those aren’t terms with which the current Microsoft regime seem to be familiar.

4 users thanked author for this post.

HiFlyer, samak, BobbyB, GreatAndPowerfulTech

Reply | Quote

Seamless

Going forward

🙂

2 users thanked author for this post.

HiFlyer, samak

Reply | Quote

progress

change

1 user thanked author for this post.

HiFlyer

Reply | Quote

Looks to me like that chart shows a declining acceptance of each new release.

2 users thanked author for this post.

Noel Carboni, SAS@HA

Reply | Quote

My thoughts exactly.  You can’t have any reported issues if you have no customers.  Very smart of them.

Reply | Quote

I think Mr. Fortin’s post would make an excellent thesis for his PHD at Baghdad Bob University.  🙂

Red Ruffnsore

3 users thanked author for this post.

lurks about, Cybertooth, HiFlyer

Reply | Quote

MS market drones wouldn’t be hired by Honest Abe’s Used Cars because Abe has some scruples.

1 user thanked author for this post.

HiFlyer

Reply | Quote

“While our measurements of quality show improving trends on aggregate for each successive Windows 10 release, if a single customer experiences an issue with any of our updates, we take it seriously… ”

Yet your actions and words over the last several years show the complete opposite.

In fact, I’d say that history indicates that the only time you take anything seriously is when the proverbial poo hits the fan and the media chastises you for it. 1809 is arguably the first time I think I’ve seen actual, real change from your company in probably a decade.

Susan Bradley’s experiment and subsequent proof showed to you months ago that things needed to change, yet you were too busy to do anything other than send several canned indifferent responses.

You have grown apathetic, and it clearly shows. So here we are.

8 users thanked author for this post.

samak, Elly, lurks about, Noel Carboni, woody, lanceboil, GreatAndPowerfulTech, HiFlyer

Reply | Quote

It’s just all the more “say it” instead of “do it”, saving much money.

-Noel

1 user thanked author for this post.

HiFlyer

Reply | Quote

“experience”. “Experience” or “experiences” was used 14 times in that post. A couple of years ago a highly-publicized marketing study reported that millenials preferred spending their money on experiences rather than possessions. By that they meant things like travel, culture, etc. Not operating systems. Someone should tell Microsoft.

2 users thanked author for this post.

Elly, HiFlyer

Reply | Quote

Maybe, just maybe…

1703-1709 (even 1607) is more because people don’t want upgrade into a unstable build.

1809 is almost zero because no-one-install-it, or at least, no one want that.

…and…

1803 is the people who jumped into the “new stable version”

perhaps?…

Reply | Quote

Hey Y’all,

Here’s the ones that really had me saying “are they really saying that?”.

“Self-host” means that employees working on Windows run the latest internal versions on their machines to ensure they are living with Windows.

The “aggressive” part refers to the tenacious push to make sure local teams run their own builds and pursue any issues found.

Really, they’re developing fixes on new builds that haven’t been integrated with all the teams builds REALLY!

We obsess over these metrics as we strive to improve product quality, comparing current quality levels across a variety of metrics to historical trends and digging into any anomaly.

Maybe, just maybe, they should be obsessing over the quality of the software and not the metrics that are so obviously failing to deliver trustworthy patches and updates.

Just my two cents worth.

May the Forces of good computing be with you!

RG

PowerShell & VBA Rule!
Computer Specs

4 users thanked author for this post.

HiFlyer, Elly, lurks about, woody

Reply | Quote

Sounds like they came up with a new definition of “dog food” — a staple of the Windows testing regime since Windows 3.1 (if not earlier).

2 users thanked author for this post.

Elly, HiFlyer

Reply | Quote

Yes, but you do not ever read the text “New Look! Now Lower Quality” on the label.

Reply | Quote

Yes, and even lesser noticed after the “New Look…!” is the other change along with that is the size change from 24 oz. to 18 oz.

But hey, it is a “New Look”!

Windows 10 Pro x64 v1909, Windows 7 Home Premium x64, Windows Vista Home Premium x64

Reply | Quote

Woody, it definitely was earlier…I remember running Windows 286 and Windows 386 (the precursors to Windows 3.1).  Those versions are what put the infamous BSOD on the map.

Windows 10 Pro x64 v1909, Windows 7 Home Premium x64, Windows Vista Home Premium x64

1 user thanked author for this post.

Elly

Reply | Quote

I bet they also believe in the Tooth Fairy!

Reply | Quote

Seems to me that graph describes declining expectations.

A social engineering success for Microsoft!

-Noel

1 user thanked author for this post.

HiFlyer

Reply | Quote

I was intrigued by the expression “complex ecosystem” in this context. I looked up “complex ecosystem”:-

“An ecosystem is a complex system composed of organisms living in a given habitat. Plants and animals are the biotic components of the ecosystem, while the subsoil, water, air, light, temperature, the climate, rains are part of the abiotic components.” (http://www.eniscuola.net/en/argomento/what-is-an-ecosystem/terrestrial-biomes/the-ecosystem-a-complex-system/)

That sounds very impressive, but unfortunately my P.C. does not fit the description. It’s a machine, bless it. Despite certain appearances, I’ve yet to find organisms in its software. It has no consciousness independent of its programmer and has yet to be caught procreating. What on earth are M/S thinking? That Windows 10 is a virtual reality universe?

4 users thanked author for this post.

HiFlyer, BobbyB, samak, Elly

Reply | Quote

Well it’s obviously too complex for Microsoft to understand and manage!  🙂

Windows 10 Pro 22H2

1 user thanked author for this post.

HiFlyer

Reply | Quote

Noel Carboni wrote:

How to use up a company’s reputation, built over a lifetime: Say things, instead of doing things. It’s cheaper! People will take a Long Time to let go of belief.

-Noel

I never mistook Microsoft for Mother Teresa, but while Gates and Ballmer were there the company did pay some attention to their public. That changed radically when Gates/Ballmer left.

GaryK

2 users thanked author for this post.

Noel Carboni, Cybertooth

Reply | Quote

Another problem with the statistics here… the burden of being beta testers lays heavily on W10 Home users… since the people I know use Home, that might account for their higher reporting of problems (everyone that I know that use W10 Home has had problems)… but Microsoft may be using their unfortunate experiences to protect a larger number of Education and Enterprise machines… thus showing less problems overall. But its killing the Home users! No one recommends buying/using W10 Home now. If you mention problems with W10 Home at Best Buy they will steer you to a Chromebook… without hesitation… they want satisfied customers.

Non-techy Win 10 Pro and Linux Mint experimenter

2 users thanked author for this post.

HiFlyer, JohnW

Reply | Quote

The other problem might be, since Home users are the unpaid and UNTRAINED (caps for emphasis) beta testers, how many problems go unreported because Home users don’t know how to report them?  There’s a reason my IT department likes me – I did tech support in a previous job, so when I email them, they know that they’ll actually get what they need to know, properly written up.  I know what to write when reporting a bug, and can track down how to do so – but I’m not an average Home user (on rare cases I use my PC).  Are average Home users up to that?

Reply | Quote

That’s why they need to spy on people so they can get the information they need rather than trust the user to report it.

Reply | Quote

The Surfing Pensioner wrote:

It’s a machine, bless it. Despite certain appearances, I’ve yet to find organisms in its software.

Well, people do keep finding bugs in Windows, don’t they?  😉

But more seriously, it grates on me too that people are using the word “ecosystem” to refer to things where it clearly doesn’t belong. Just call it (as the case may be) a “market,” an “industry,” a “company,” an “OS,” what have you. All these nouns are sufficient to describe what they refer to. There’s no need to get figurative about what you’re taking about. Saying “ecosystem” is lazy, sloppy thinking.

And if you want to use the most all-encompassing, general term that you can, then for heaven’s sake just say “system” and save yourself (and us) two syllables. You sound neither hip nor knowledgeable when you say “ecosystem”.

 

Keep Running Windows 7 Safely for Years to Come  
Make Windows 10 look and work like Windows 7

Reply | Quote

“Ecosystem” is a good long word to use for impressing the rubes. Which, in the opinion of the PR people who use it, is what the vast majority of us are. It is up there with “momentarily” used to mean “shortly” or “soon” (but Yuck! those are short words). We live in a pretentious age where self-important people use long words to impress others and each other.

Ex-Windows user (Win. 98, XP, 7); since mid-2017 using also macOS. Presently on Monterey 12.15 & sometimes running also Linux (Mint).

MacBook Pro circa mid-2015, 15" display, with 16GB 1600 GHz DDR3 RAM, 1 TB SSD, a Haswell architecture Intel CPU with 4 Cores and 8 Threads model i7-4870HQ @ 2.50GHz.
Intel Iris Pro GPU with Built-in Bus, VRAM 1.5 GB, Display 2880 x 1800 Retina, 24-Bit color.
macOS Monterey; browsers: Waterfox "Current", Vivaldi and (now and then) Chrome; security apps. Intego AV

2 users thanked author for this post.

HiFlyer, Cybertooth

Reply | Quote

Ecosystem’s a good analogy for software development, especially at the scale Microsoft works.  As software “evolves” it creates new pressures within the ecosystem that other software devs react and adapt to.  You have evolutionary lines (eg, Microsoft Office), and you have dinosaurs (around the 80s).  For larger applications, code constantly has to change to keep up with the times and make sure that everything stays working within its ecosystem.

So no, your computer isn’t alive, but it has continuously changed.  Generally for the better, but sometimes like now for the worse.

 

Reply | Quote

I suspect that Microsoft’s filings with the Securities and Exchange Commission (Forms 10K, 10Q and 8k) do not adequately disclose the dissatisfaction and problems with the Windows update system.

Reply | Quote

anonymous wrote:

“We shifted the responsibility for base functional testing to our development teams in order to attempt to deliver higher quality code from the start, but often fell short as exemplified by the numerous issues both business and end users have experienced including, but not limited to, data loss.” There, fixed THAT sentence.

When the jargon starts, truth is usually the first casualty.

GaryK

1 user thanked author for this post.

HiFlyer

Reply | Quote

I think the graph shows that people stopped complaining over time because they already know that their complaints won’t be heard.

2 users thanked author for this post.

RamRod, HiFlyer

Reply | Quote

M$ multi-syllable bafflegab reveals more than it conceals.  Pitiful!

Words of one syllable work.

 

Reply | Quote

Netmarketshare numbers for Windows desktop OS:
Sep 2016 = 90.45%
Oct 2016 = 89.97%
Oct 2017 = 88.00%
Oct 2018 = 87.27%

Since the cessation of the 1-year free upgrade(from Win 7/8.1 to Win 10) promotion in Aug 2016, Windows has lost more than 3% in world marketshare = about 60 million desktop/laptop computer users left Windows = these ex-Windows users just up and leave = not feed-backed or recorded by M$ or the numbers were just ignored by M$.

The above numbers likely prove that Win 10 patching has gone down the drain. […]

2 users thanked author for this post.

Cybertooth, HiFlyer

Reply | Quote

We also changed the focus of the teams that still report to me who are responsible for end-to-end validation,

My wife uses Win10 Home.  Does she now report to Mr. Fortin?

And, if so, what’s she pulling down, after taxes?  Just curious.

1 user thanked author for this post.

HiFlyer

Reply | Quote

woody:

check out the reaction from Ed Bott from this ZDNet article:
https://www.zdnet.com/article/microsoft-resumes-rollout-of-windows-10-version-1809-promises-quality-changes/

and from Bogdan Popa’s Softpedia article:
https://news.softpedia.com/news/microsoft-s-windows-10-version-1809-mea-culpa-doesn-t-mean-anything-523784.shtml

2 users thanked author for this post.

AlexEiffel, HiFlyer

Reply | Quote