Windows Blog: “Data, insights and listening to improve the customer experience”

Topic

New Reply

Yesterday, Rob Mauceri and Jane Liles published a white paper on the Windows Blog that talks about using telemetry to figure out if a patch is ready f
[See the full post at: Windows Blog: “Data, insights and listening to improve the customer experience”]

5 users thanked author for this post.

Myst, abbodi86, Karlston, Elly, GreatAndPowerfulTech

Reply | Quote

Replies

Are the gamers “seeking” these second updates?  Because if you don’t “check for updates” [which isn’t check for, it’s install them because I’m insane] you won’t see these patches pushed to you.

It gets back to insiders are not real world testers.  They are many times on VMs and test machines.  Heck my insider is on a surface I don’t use on a daily basis.  You have to USE a PC to SEE issues.  Nothing is a better test bed then a crusty used pc.

Susan Bradley Patch Lady/Prudent patcher

14 users thanked author for this post.

Myst, Karlston, Elly, woody, wdburt1, Chronocidal Guy, johnf, fk5353, CADesertRat, SteveTree, Microfix, Steve S., satrow, GreatAndPowerfulTech

Reply | Quote

As someone who uses their PC primarily as a niche gaming console (mostly for flight simulations), updates are the last thing I want to put my system through.

I’d actually wager that Windows 10 is probably the last thing any dedicated gamer wants on their custom gaming rig.  The analogy that comes to mind for me is that highly tuned gaming PCs are similar to heavily customized muscle cars.  The last thing any owner wants is a mandate that their heavily tuned car will be sent back to the manufacturer for a “tune up.”  They know the techs will take one look at the engine, see all the customized bits, and go “This isn’t right, we have to rebuild this with stock parts!”

I would honestly never expect Microsoft to ever gather any useful telemetry from gaming systems.  I would expect gamers to postpone any updates as long as humanly possible, to avoid the potential disruption to their hobby.  They know their systems, they know Microsoft’s history, and the last thing they want is an update mucking up the works.  Consequently, Microsoft just isn’t going to get any meaningful feedback on how their patches function on that massive pile of corner case configurations.

(Apologies if this is a double post, seems like the previous one self-deleted when I tried to edit it.)

6 users thanked author for this post.

lanceboil, Myst, GoneToPlaid, Elly, wdburt1, Rock

Reply | Quote

Your post was caught in the spambucket because of too fast submit-edit-submit. Slow down and give the system time to process the submit before you try to edit.

Reply | Quote

Good analysis.  I am not a gamer, but long ago I found out that if you really want to learn about computer monitors and keyboards, check out what gamers have to say.  There is some overlap with the concerns of those of use who want to use the computer for so-called “productivity.”  The two groups with the greatest interest in N-key rollover, for example, are gamers and… really fast typists.  So–gamers are far from alone in regarding their setups as finely-tuned machines that should remain undisturbed unless and until the owner sees a really compelling reason to stop productive work and upgrade the tools.

 

1 user thanked author for this post.

GoneToPlaid

Reply | Quote

I agree. Gamers are good sources for info about what motherboards, brands of memory, monitors, and other stuff which offer better performance for the buck. Gamers were the first to document the flaws in Intel’s first releases of new microcode updates to address Meltdown and Spectre. They quickly discovered, on non-overclocked computers and even on underclocked computers, that Intel’s initial microcode for Meltdown and Spectre was causing CPUs to throw a ton of recoverable errors and occasional non-recoverable errors. The latter could result in instant reboots and/or either loss of data or corrupted data.

I also agree with your final remarks about not stopping the productive experience. This is something which Microsoft has been miserably failing at, as a result of what seems to be an unending barrage of botched updates. I’ll stop now, before I start to rant.

1 user thanked author for this post.

wdburt1

Reply | Quote

Of course there’s also the other side of it…

It’s common for gamers have one-off machines with lots of tweaking and adjustment functions, and some of those setups only become worthwhile *if* you can tweak it to behave just right. So, manageability features tend to be sort of low priority for them, strict correctness of results is secondary to a smooth experience, and even repeatability might be in question.

(Sheesh, testing GPU offload floating-point calculations on a gaming-type GPU vs a “pro” type was … revealing… at least that one model clearly sacrificed accuracy to gain speed.)

So yeah, significant biases in a sample set if collected from there too.

Reply | Quote

The gaming culture from forums such as nvidia’s and many reddit forums tends to push updates as a solution for games not running correctly, as in:

issue: “my X game stutters on my 2180 ppx firecrazy card(tm).”
solution: “You need to update windows and drivers and try again”.

Reply | Quote

It seems they believe their own ‘Microsoft-speak’: that it’s all about good customer experience. But without real-world, daily use scenarios, the telemetry results end up being an inadequate indicator of the software’s true readiness.

What the forced updates and telemetry appear to be in actuality, is an effort to externalize costs onto the shoulders of the consumer – primarily for the benefit of MS profitability and the top tier of customers: the enterprises. Not to disparage enterprise customers, BTW. <rant finished>

Win10 Pro x64 22H2, Win10 Home 22H2, Linux Mint + a cat with 'tortitude'.

1 user thanked author for this post.

GoneToPlaid

Reply | Quote

The entire telemetry thing was Microsoft’s attempt to become the most powerful advertising powerhouse in its war with Google. I decided to read the link in the article which Woody linked to. To save everyone time, here is the link:

https://support.microsoft.com/en-us/help/4468236/diagnostics-feedback-and-privacy-in-windows-10-microsoft-privacy

Oh my. The entire telemetry thing now sounds so benign in comparison to how telemetry was originally envisioned by Microsoft to serve up targeted advertising directly onto every Windows 10 user’s computer, so that Microsoft could pursue its war against Google. It seems that the days of targeted advertising may be numbered. Many companies are belatedly discovering that cheaper and non-targeted advertising not only costs less, but also is generating more revenue for them. Now I have to stop, before I rant!

3 users thanked author for this post.

Steve S., AlexEiffel, wdburt1

Reply | Quote

Having a methodology and effectively using it are two different things.  Starting with Win 10, and now affecting all current versions of windows, effective use of the methodology seems to have gone by the wayside.

Reply | Quote

That is how I see it. I feel we have moved into “so clever, it’s dumb” with machine checking on machine as regards patching. Get the OS changes out to a broad based group who use real computers, in the real world, for a while and then sort the problems. Can’t be that difficult (yes I know).

Over-reliance and bluster about AI/VM testing ….. blah di.. blah di ….blah is just smokescreen about getting rid of human engineers…….. AI don’t raise issues like people……. Yet!

Real people testing real software in real workplaces before general release…… Why on earth would that work?

1 user thanked author for this post.

GoneToPlaid

Reply | Quote

Uh, that did work rather well for Microsoft and for years — up until Nadella fired the windows update quality control team. I miss the days of not worrying too much about installing updates. In the olden days (pre-Nadella), there were no Rollups. If a particular update was found to have issues, one could simply uninstall the single offending update. Now, it is all or nothing with any given month’s updates, since individual updates are packaged into a Rollup. Given that all updates are Rollups, and given that Microsoft now longer has a windows update quality control team who used to put all individual updates through a serious vetting process, all of us have to deal with botched updates virtually every single month. This is Nadella’s Customer Experience.

Reply | Quote

I am truly proud that Microsoft will continue to keep doing exactly what they’ve been doing because it has worked so well so far! I count four uses of the word “quality” in that Microsoft marketing-speak paragraph quoted above. Too bad there’s virtually no quality in the actual updates.

It’s been, what, at least 8-9 months solid of bad patches, rereleased patches, extra patches, and bug after bug after bug: the perpetual Japanese calendar problem, the Documents folder being eaten by 1809, Access 95 database glitch, game slowdowns this month, service outages, and on and on. It is difficult for me to keep up, I admit. I have a vague recollection of one month being okay–maybe it was December–the one where Microsoft shut down for the holidays. That’s got to be a coincidence… Right?

3 users thanked author for this post.

lanceboil, GoneToPlaid, Karlston

Reply | Quote

Amen Bro, MS has failed to successfully substitute use of telemetry for adequate in-house QA testing prior to release of patches/updates. MS has always implied that the new testing regimen involved using average home users as “unpaid beta testers” and using telemetry to catch problems before releasing the bits to the enterprise ring. So what if a few unsuspecting souls end up paying a couple hundred bucks to get back a borked system as a result. After all, it was only a small proportion that was adversely affected. I really enjoy months like February where parade of ongoing bugs, if acknowledged, is fixed in a preview release that needs to be installed to address the issue. That’s really sweet as you are being told to install a preview which to me is tantamount to being a beta tester. Honestly, am I the only person that gets annoyed just reading the c**p that MS publishes these days?

2 users thanked author for this post.

GoneToPlaid, Karlston

Reply | Quote

The fatal flaw of any statistics driven analysis is the relationship of the sample to the population. If the sample is truly representative of the population then statistical analysis is useful and will give valid results. The problem is how is the sample selected. It does not appear to be a truly random sample but one biased towards home users who are not using bleeding edge hardware and are doing relatively mundane tasks with their boxes. The bias is hard to correct for even if you know it exists. Telemetry is biased by design to oversample the unsophisticated home user over the more technically savvy user.

A real QA department that actually tests builds on real equipment could have gaming rigs setup for this purpose. But since MS does not have a proper QA department they cannot do this type of testing.

2 users thanked author for this post.

witp, GoneToPlaid

Reply | Quote

Absolutely – Microsoft payroll using Microsoft AI isn’t ever going to replicate real life usage.

Reply | Quote

You have totally hit the nail on the head, and not just in terms of how the statistical sample is collected and that it is not a truly random sample. The real problem is that AI is nowhere advanced enough to be able to test all real world user scenarios — let alone being anywhere close to simulating most real world scenarios. Given this fact alone, what Microsoft chooses to gather in terms of statistical samples is pretty much irrelevant. If the AI always spits out junk in terms of supposed quality because the AI is inherently deficient in its capabilities, then it really doesn’t matter what statistically cherry picked junk is fed in.

I don’t think that Microsoft will ever be able to recreate a new windows update quality control team. Once they were fired, they all went to other jobs. I bet that it will be cold day in Hades before any of them choose to come back. I have a feeling that a lot of other people jumped ship as well. The exact same thing happened to NASA when Obama redirected NASA’s mission to be one of education instead of exploration.

1 user thanked author for this post.

lanceboil

Reply | Quote