Can you quantify the damage done by bad patches?

Topic

New Reply

I’ve long concentrated on explaining patches’ bugs and how to work around them. Some people think that patches in the last 20 years don’t break stuff.
[See the full post at: Can you quantify the damage done by bad patches?]

Reply | Quote

Replies

Still remember some cases I’ve seen in the internet, where Firmware Updates from Microsoft broke their Surfaces – so users had to send the devices back to get a replace.

https://www.google.com/search?q=surface+firmware+update+bricked+device&oq=Surface+firmware+update+bricks&aqs=chrome.1.69i57j33.11696j0j7&sourceid=chrome&ie=UTF-8

Ex Microsoft Windows (Insider) MVP, Microsoft Answers Community Moderator, Blogger, Book author

https://www.borncity.com/win/

1 user thanked author for this post.

woody

Reply | Quote

Without any doubt. Lost assignments because of bodged updates, repushing them after restoring images, fighting with ‘broken’ hardware after updates and upgrades. It’s why we left Windows 10 and moved to Macs (Office) and Linux. At least there you still have the free choice about when to upgrade. Since then, no [annoyed] clients anymore and no long nights trying to control or repair the damage done.

1 user thanked author for this post.

woody

Reply | Quote

I hear ya… but can you quantify the damage?

Reply | Quote

[Alex5723]

If you look at damages due to ransomware my opinion is the Microsoft’s update damages are 100x+.
We are talking $$T yearly.
Microsoft’s updates are installed monthly while ransomware is really rare on yearly basis.

The Cost of Ransomware

In the past year, there has been a 235% increase in cyber threats targeting businesses, and ransomware is an increasing risk for organizations of all sizes.

A recent report from Malwarebytes says that business ransomware attacks increased 365% from Q2 in 2018 to Q2 in 2019.

The Cost of Ransomware

Ransomware damages were expected to exceed $11 billion in 2019.

Kaspersky Lab discovered the average enterprise pays more than $1.2 million per attack, with small businesses paying about $120,000. For both enterprise and small businesses, costs continue to increase year over year. Kaspersky says one successful ransomware attack can cost an organization more than $713,000. In addition to the ransom other associated costs can include:

Data loss or damaged data
Loss of business functions and downtime
Loss of sales and/or production
Cost of investigation
Damage to brand and business reputation
Expenses related to system repair and restoration

COSTS RELATED TO DOWNTIME

On average, downtime costs can exceed five to 10 times the amount of a ransomware payment. In mid-2019, the average payment was more than $36,000, with many organizations paying more than $100,000.

To quantify ransomware downtime in terms of days, the average loss is 9.6 days….

• This reply was modified 4 years, 7 months ago by Alex5723.

1 user thanked author for this post.

Mr. Austin

Reply | Quote

That’s my impression, too, but I have no hard facts to back it up.

I’m surprised this topic hasn’t garnered more reports like “My company went down for a day” or “I had to reinstall Windows from scratch and it took me all day.”

1 user thanked author for this post.

doriel

Reply | Quote

“My company went down for a day” or “I had to reinstall Windows from scratch and it took me all day.”

This is very close to the reality. My last real enterprise outage was caused by Windows update on CVE-2020-1472. We had to rebuild completely new Domain Controller and we spent approx 5 hours on it (me and my two celleagues). We had to troubleshoot exchange server as well, because of that update.

So for me the quantification is clear. Its costing human time, thus costing money since IT guys are payed for that.

And I have several notebooks from my customers, who has problems with updating. They can quantify it clearly – the ammount of money they pay us for the time we spend on their notebooks.

Dell Latitude 3420, Intel Core i7 @ 2.8 GHz, 16GB RAM, W10 22H2 Enterprise

HAL3000, AMD Athlon 200GE @ 3,4 GHz, 8GB RAM, Fedora 29

PRUSA i3 MK3S+

Reply | Quote

[Alex5723]

I don’t think that Enterprises are allowed to disclose publicly the amount of damages caused by Microsoft’s updates under their Enterprise license agreement, hence no surveys/papers on the subject.
Microsoft has been sued by individuals, in most cases after forced Windows 10 upgrades.

• This reply was modified 4 years, 7 months ago by Alex5723.

1 user thanked author for this post.

Cybertooth

Reply | Quote

Good tweet by Florian Roth:

I‘d recommend a series of conversations with admin teams of big companies about problems caused by early patching & how they value availability. It’s a risk assessment: is the risk of outages or unexpected service behavior due to patching higher than the risk it tries to address

Reply | Quote

I’ve had patches that caused Access crashes, or incompatibility with certain spreadsheets.  Had to roll them back.

I’ve had patches that caused Excel crashes when opening files. Had to roll them back.

I’ve had patches that caused Outlook crashes on opening. Had to roll them back.

I’ve had Windows 10 updates that caused multiple random systems to stop printing properly; it took Microsoft a week to resolve.

I could also tell you of our firewall vendor releasing an update that bricked firewalls by updating GRUB and breaking it, putting the OS in rescue mode and requiring reload of the entire firewall. I could note that our vendor defaults their firewalls to auto-update, and we were lucky they caught it quickly, because I lost only two firewalls -too bad they were for the same client, who lost an entire day’s productivity.  The vendor very quietly swept that under the rug with minimal notice to customers, eroding confidence; they didn’t help by releasing an automatic emergency update that overrode automatic update preferences and updated firewalls anyway, sometimes spontaneously restarting them mid-day, without notifying customers (thankfully I didn’t end up experiencing that fiasco).

So absolutely patches do break things and cause problems, including monetary loss. There’s just little way to hold a vendor accountable other than switch, and how do you switch when the vendor is Microsoft?

We are SysAdmins.
We walk in the wiring closets no others will enter.
We stand on the bridge, and no malware may pass.
We engage in support, we do not retreat.
We live for the LAN.
We die for the LAN.

Reply | Quote

Well this is debatable whether it was a good or bad patch!
The now redundant (and removed) KB3035583 in conjuction with various other patches, caused untold grief for the masses. Luckily askwoodydotcom came to the rescue for us (pre-forum era)

Windows - commercial by definition and now function...

Reply | Quote

woody wrote:

Some people think that patches in the last 20 years don’t break stuff.

Some people know that patches in the last 20 years haven’t broken any of their stuff.

The only way you can quantify “bad” patches is (first) as a percentage of those for whom patches have had no ill effects, and (second) is the “broken stuff” a result of a single patch, or the culmination of applying a patch to a system that was not fully patched to begin with.

There are few difficulties posted within these forums for which I can offer corrective measures, because my first question of someone with difficulties is, “Are you completely up to date with updates/patches?” which is, for the most part, out of place at AskWoody.

But, a total number of folks within the category you’ve lined out has no relevance unless that number is placed within the greater context of those folks who have not had any problems.  Otherwise there is no comparison.

Always create a fresh drive image before making system changes/Windows updates; you may need to start over!

We all have our own reasons for doing the things that we do with our systems; we don't need anyone's approval, and we don't all have to do the same things.

We were all once "Average Users".

Computer Specs

Reply | Quote

[Mr. Austin]

In essence you’re asking if someone (a company or companies) has audited the extent of Microsoft’s monetary damages aggregated from categories which include diversions of valuable staff time, consultants’ paid time, damage to software or hardware, and the infrequent but not unheard of bricking of some devices. In principle I feel that someone exists who could give you decent quantifications. Long ago I happily used PatchLink on our comparatively modest, mixed OS LAN, and there will be more current analogues. Check Point technologies and other companies which need to monitor Microsoft also come to mine. As long as such a company isn’t so frequently in bed with Microsoft that they’ll make more money from that relationship, rather than by being a mostly impartial organization, like Interpol.

Human, who sports only naturally-occurring DNA ~ oneironaut ~ broadcaster

• This reply was modified 4 years, 7 months ago by Mr. Austin.

Reply | Quote

.Net was the last horrible one
https://redmondmag.com/articles/2018/07/23/microsoft-pulls-july-net-patches.aspx

 

 

Reply | Quote

You are asking about the emperor’s new clothes. Most companies are unwilling to admit to operating system vulnerabilities or software difficulties. If documented, difficulties remain an in-house report and not subject to review by outsiders.

On permanent hiatus {with backup and coffee}
offline▸ Win10Pro 2004.19041.572 x64 i3-3220 RAM8GB HDD Firefox83.0b3 WindowsDefender
offline▸ Acer TravelMate P215-52 RAM8GB Win11Pro 22H2.22621.1265 x64 i5-10210U SSD Firefox106.0 MicrosoftDefender
online▸ Win11Pro 22H2.22621.1992 x64 i5-9400 RAM16GB HDD Firefox116.0b3 MicrosoftDefender

Reply | Quote

I’ll never forget the first version of Windows XP had the firewall turned off by default-

Reply | Quote

Microsoft should be able to answer this question – accurately and precisely. What do you think they do with all of the telemetry they collect? What is the main purpose of the telemetry? They know exactly how much damage their updates do – and then they measure if they should pull it back or revise it – and how fast they need to work to revise it.

Or they could simply test it better before deploying it. They must’ve calculated over the years that ex post facto revisions to updates was more effective than extensive beta testing pre-deployment. I bet they have the data to support that. I wonder if they ever published it?

There had to be a reason to dismiss all of the beta testers and go with telemetry. Dr. Watson on steroids.

RamRod

Reply | Quote

Very good opinion. Telemetry should be used for this purpose. It cant evaluate money, but it can evaluate time and it can also count errors. Then we can have success percentage. But this is just raw data, how many people are not sending feedback?

There had to be a reason to dismiss all of the beta testers

Sure there was. To save money of course. There is nice sounding fairy tale:
You can join Windows 10 Insider ring and get in touch with cutting-edge technology!

The truth is:
We need you to betatest our OS. Sorry you payed money for it, but that is the reality.

In gamimg industry, testservers and sandboxes are mostly for free. Very often you get all ingame items unlocked, so you can test them. This seems to me as the correct way of treating customers, but this seems to be utopia/unreal with Microsoft. So unethical to me.

Dell Latitude 3420, Intel Core i7 @ 2.8 GHz, 16GB RAM, W10 22H2 Enterprise

HAL3000, AMD Athlon 200GE @ 3,4 GHz, 8GB RAM, Fedora 29

PRUSA i3 MK3S+

2 users thanked author for this post.

Cybertooth, LoneWolf

Reply | Quote

A forced “upgrade” to Windows 10 ended up frying the BIOS on a family member’s motherboard. Ended up having to install a new one.

1 user thanked author for this post.

rontpxz81

Reply | Quote

Yes, I can quantify the damage done by bad patches?

In fact, it has been nominal. I have worked for and/or run large corporations, government agencies, news organizations, and for the last 20 years my own firm.

At no time have we experienced a significant economic loss or loss in productivity as a result of bad Windows updates or bad patches. The only major computer related economic loss and loss in productivity that I have experienced was associated with software coding during Y2K.

Our biggest problem has been with third-party “free software” recommended on Woody’s site. As a result, we no longer use off brand software featured on the website.

At the present time we are operating 40+ Windows 10 PCs. The machines are high-end, off the shelf Lenovo’s and more recently HP’s. The only modifications we make to our computers is to install larger drives and additional RAM. All of our new drives are made by Western Digital and RAM compatible with the specifications of the manufacturers. We made the transition to Windows 10 at year-end 2019. Prior to 2019 all of our PCs were operating with Windows 7 or its predecessors without incident.

We update Windows 10 consistent with Woody’s MS-DEFON warnings as well as driver and bios updates when push to us by the system manufacturer.

Our primary software includes: Acronis True Image, Adobe Acrobat, CCleaner, the manufacturers suite of software shipped with each machine, graphics software, Logitech software to support our video cameras, various versions of Microsoft Office, Firefox, our VPN’s software, drivers and software for video cards, security software, MATHLAB, and the current WordPerfect office suite of programs.

All of our computers are automatically backed up to a second internal drive twice a day by Acronis. In addition, all of our computers are backed up to external drives that are stored off-site at the end of each business day.

In those instances when a Windows update or a patch has a negative impact on our systems, we immediately attempt to restore the system from a restore point or, worst case scenario, an Acronis backup. Time lost in people hours and the economic cost in dollars is nominal – but the machine may be off-line for a couple hours during recovery.

So, the damage done to our firm and my previous employers due to bad Microsoft updates and patches has been nominal.

Public corporations are required to report to their stock holders any  event that will have a significant impact on their operations and/or profitability.

And with over 900 million Windows 10 computers operating worldwide today there would be an absolute rebellion if Windows updates or bad patches were having a significant impact on corporate productivity and/or profitability. And such disruptions would be fully covered in the business press.

 

1 user thanked author for this post.

bbearren

Reply | Quote

You’ve mitigated damage done by bad patches by knowing how to repair the damage.  Downtime in people and machines have definite costs, costs that are known and tracked.

But, unless a machine or multiple machines are directly responsible for generating revenue or verifiably caused a quantifiable business loss, the veracity of downtime losses are questionable.  What gets reported ranges from nothing through vague to clear software caused losses.

Don’t understand how utilities recommended in Ask Woody could have been found by enough employees to cause big problems.  Good site but I’ve never seen any utility mentioned here unique to the site; they appear in many sites.  Hard to trace back to Woody.

What is nominal?  MS, with market cap that puts them well within the list of top 10 countries’ GDP,  definitely won’t publish accurate numbers on users’ lost time; they’re not responsible for it per EULA and they don’t seem to know what to do with telemetry.  Client companies, required or not, won’t report specifics; it would be stupid to reveal details of problems competitors can read in a Quarterly.

CCLeaner?  Huh? Avast is a data broker, take a look at their corporate site.  Read their financials; they’re very honest about what they do.

3 users thanked author for this post.

doriel, Alex5723, Cybertooth

Reply | Quote

Little bit out of context, but look at these sentences. Even if Windows somehow manage to ruin your bussiness, you will not get a single penny. And somehow I feel that this correct, bacause otherwise there would be thousands of people blaming Microsoft for ruining their bussinesses. The worst thing for me is the “unreliability” of this operating system and constantly changing envirinment with no obvious roadmap/ goal.

Windows 10 EULA:
“The device manufacturer or installer warrants that properly licensed software will perform substantially as described in any Microsoft materials that accompany the software. If you obtain updates or supplements directly from Microsoft during the 90-day term of this limited warranty, Microsoft provides this limited warranty for them.”

-90 day limited warranty

And another one:
Except for any repair, replacement, or refund the manufacturer or installer, or Microsoft, may provide, you may not under this limited warranty, under any other part of this agreement, or under any theory recover any damages or other remedy, including lost profits or direct, consequential, special, indirect, or incidental damages.

Dell Latitude 3420, Intel Core i7 @ 2.8 GHz, 16GB RAM, W10 22H2 Enterprise

HAL3000, AMD Athlon 200GE @ 3,4 GHz, 8GB RAM, Fedora 29

PRUSA i3 MK3S+

2 users thanked author for this post.

Cybertooth, LoneWolf

Reply | Quote

One of the earlier upgrades of Win10 took ouit two of my most used programs; Firefox and Thunderbird. It also deleted all of my folders and files. It took all of my backup drives and $370 at the repair shop to get files and folders back and had to reinstall the programs. One of my other non-MS doesn’t work any longer. It is also a graphic program like Adobe. It seems Microsoft doesn’t want any competition for their bloated win10 system. I’m not a computer graduate soi Microsoft should be fixing problems not creating them, and they seem to want those users to do that job too. I bought a cheap Chromebook hoping that will do what I need to do, rather than spend hours figuring out how to fix win10 issues.

Reply | Quote

As a home user with modest requirements, I can not recall any patches causing major damage except for the recent HP Omen Computer “problem” which repeatedly bombed my computer. But I am not sure that was caused by a patch or some other sort of update and whether HP or MS was responsible.

What I can say is that I have had repeated instances of updates causing my screen colors and arrangements to change dramatically (update to 2004 is the latest to do this), my file explorer settings to change, my wireless network to fail, and my Internet browser to change its setup, home page, and even which browser was the default. While none of these cost me real money to repair, they did cost me much time and effort to correct the damage.

1 user thanked author for this post.

Cybertooth

Reply | Quote

Since my introduction to windows 3, I have no way of calculating the number of man hours put in reinstalling, installing and troubleshooting issues caused by msft updates/upgrades/patches etc etc etc. But in light of the fact my hrly wage was north of $75 and time is money, I should guess msft owes me at least 4 first steps of children, uncounted athletic events my children and grandchildren were participants in, family reunions(not to be confused w/ PC help) hrs and hrs on telephone both conversing and on hold, miles driven to replace damaged hardware, gasoline, auto repairs, shipping and handling, searching online,- ad nauseum….I would say at least 46,9567,898 USD + tax.

I am obviously not a business user BUT my time is as valuable as anyone elses- including Billy gates’.

PS Not to mention University classes mucked up by shoddy coding by MSFT.

Reply | Quote

I am surprised to see that there are some people that think that Windows Patches are perfect

but well,   There’s a reason why the Msdefcon system exist here and why most people should pay attention to it….

2 users thanked author for this post.

Cybertooth, rontpxz81

Reply | Quote

No, Windows patches are not perfect.

But the question on the table is, “Can you quantify the damage done by bad patches?”

Within the corporate environments where I have worked, Windows updates and patches have occasionally brought down a PC. But because of the ability to recover through the use of restore points or backups the economic/productivity damage caused by Windows has been nominal.

Reply | Quote

F A Kramer wrote:

they did cost me much time and effort to correct the damage.

If you value your time by 1 hour = $500 and add the effort…how much it “didn’t cost you in real money” ?

Reply | Quote

By “real money” I mean cash out of my pocket to pay someone to repair the damage, or for the replacement of a computer. For that the answer is none. In “virtual money”, it is hard to say. Could I have been earning real money during the time spent undoing MS “revisions”? Perhaps, but even in my dreams, not $500/hour. Not even $50. But whether the expense is in real money or not, the need to undo MS damage should have been unnecessary and I think that is the point most of us are making.

Reply | Quote

The question posed by Woody at the beginning of this thread is, “Have any patches in the past five years caused you or your company “considerable damage”?”

In our case it has not.

We have developed and put into place a written emergency recovery program for our firm. Its chapters include, but are not limited to, addressing power failures and power surges, a fire, computer related issues, storm related damage, pandemic response, etc.

The computer section includes hardware redundancy, data integrity and preservation, data and system recovery, an inventory of spare components including hard drives and monitors, etc.

As a result, if a computer goes down we can pull a backup, fully configured, unit off the shelf, plug it in, recover the data files from the damaged PC, and be back to work in minutes.

As a result of our compliance with our emergency recovery program we have not experienced “considerable damage” from bad patches in the last five years.

2 users thanked author for this post.

doriel, bbearren

Reply | Quote

In our company we are protected against data storm for example, but against patches? We have WSUS

We have developed and put into place a written emergency recovery program for our firm.

This enterprise risk management should be standard, good job. But what about home users? 0% has power backup source. Lets say that 10% of average users understand the principle of backing up whole partitions. Are they able to set up backing software correctly? Of course not, and why should they? They just want to use their computer. They have no intention in this patching madness.

And home users are the MOST affected, cause they do not backup regullary. These users backup just their data – movies, photos, documents. So their recovery does not take 10 minutes, but they struggle and go to their PC guru and it takes days and money to get their PC back.

Dell Latitude 3420, Intel Core i7 @ 2.8 GHz, 16GB RAM, W10 22H2 Enterprise

HAL3000, AMD Athlon 200GE @ 3,4 GHz, 8GB RAM, Fedora 29

PRUSA i3 MK3S+

1 user thanked author for this post.

Cybertooth

Reply | Quote

Kathy Stevens wrote:

The computer section includes hardware redundancy, data integrity and preservation, data and system recovery, an inventory of spare components including hard drives and monitors, etc. As a result, if a computer goes down we can pull a backup, fully configured, unit off the shelf, plug it in, recover the data files from the damaged PC, and be back to work in minutes.

For me, this is the bigger question.  Why isn’t everyone doing the above in some proportion relevant to their needs and resources?  I have a spare 2.5″ SSD, mSATA SSD on standby for my daily driver desktop, and three months worth of protected drive images at the ready.  I have a formatted 4TB HDD on standby to swap into my NAS should one of its four drives fail.

I’m an inveterate tinkerer and regularly plumb Windows innards.  I can spend a good deal of time moving and breaking things to the point that Windows no longer boots, but it only takes 6 minutes to completely restore with a recent drive image.  It only requires imaging software and an external storage drive for that level of protection.

If an ounce of prevention is worth a pound of cure, a pound of prevention can be invaluable.

Always create a fresh drive image before making system changes/Windows updates; you may need to start over!

We all have our own reasons for doing the things that we do with our systems; we don't need anyone's approval, and we don't all have to do the same things.

We were all once "Average Users".

Computer Specs

1 user thanked author for this post.

Kathy Stevens

Reply | Quote

One closing thought.

No, patches have not caused considerable quantifiable economic damage to our company in the past five years.

But there is the intangible cost of having to manage Windows 10 since migrating our systems from Windows 7 at the beginning of the year.

Time spent following the literature to find out what are the potential problems associated with patches and updates, when is it safe to patch and update, what patches to avoid, etc.

Time better spent dealing with our clients and doing analysis.

1 user thanked author for this post.

Cybertooth

Reply | Quote