Is it OK to run patches on 500+ VMs?

Topic

New Reply

Just saw this message from ME: I haven‘t approved updates since 12/2017 for our infrastructure with 500+ VMs. I‘m not new to that topic but your team
[See the full post at: Is it OK to run patches on 500+ VMs?]

5 users thanked author for this post.

geekdom, ve2mrx, Elly, Laptop Boy, WildBill

Reply | Quote

Replies

What I have done in this occasions is to approve them in batches.

I like to go by year, in that way you can see what you are approving (don’t want to deploy .net 4.7 to web servers if that is not your intention)

Also you can monitor what systems are running low in space in that way you don’t stop exchange from working due to lack of space.

take this time to go as far back as you can to see if patches from 2015 are applicable (u might have 2 systems)

When I do catch-up on clients I do it one batch a week (200 servers and 3000 WKS on average) aprox… that way is easier to troubleshoot if things happen.

 

Reply | Quote

What operating systems?

1. Have canaries.  Machines in your office where the users are savvy and use sample representations of your software.  They get it first.  They report back to you if anything broke.

2.  Have a rule that all patches get installed by X date.  Honey if you haven’t installed anything since December, we need to talk.  I go no more than 30 days.  I do not install on day one.  Or even day five.  Or day 10.  At the earliest it’s day eleven (on the second weekend after patch Tuesday).

3.  Standardize on machines/hardware.  One thing the 10 era has documented – one thing the Spectre/Meltdown has showcased is that we need to include bios and drivers in our patching sequences.  A lot of issues are cleared up with newer drivers.

4.  I’m on the prior months updates by the time we enter the new month’s patching sequence.  So right now May updates are installed.  If you can’t install the prior months updates right now — what specific patching side effect are you worried about?

Email me/private message me and let’s discuss as to why you are not comfortable and what options we can do to get you comfortable.  As 500 machines with no updates since December scares me more than updating through May.  And that’s the key …when I get to the point that I’m more scared about being unpatched than I am about patching that’s the point where I’m ready to patch.

Susan Bradley Patch Lady/Prudent patcher

6 users thanked author for this post.

AlphaCharlie, ve2mrx, Elly, AlexEiffel, Noel Carboni, Bill C.

Reply | Quote

I think there’s something to be said for caution.  May is looking good and is what I’ll be moving my servers up to (my desktops are all on May already) from December 2017.  Before April, December 2017 was the last time patching was stable enough to be considered good.  January’s a good example with its TotalMeltdown exploit that took until March for it to be addressed, only to be released in a slew of patches that also broke virtual and Intel NICs.

It might be a taste of things to come when outdated machines are more secure than bleeding-edge systems, especially as Win10/Server2016 becomes a service.

3 users thanked author for this post.

Elly, Noel Carboni

Reply | Quote

To Patchlady Susan.

This is a great reply, and actually quite applicable even though I am not an admin or corporate IT person overseeing servers and dozens of machines, to the individual user.  As a home user with a few machines (plus some friends seeking advice) I found your first and second points are very applicable to the home user, especially if using a desktop.

On Point #1.  My canary is my main machine, but that is in reality a second line canary.  The first line is actually the MVPs and participants in AskWoody.com and your patchlist, which I have used on and off for a number of years, before you were here.  My main machine must remain operational, so I use the month after patch Tuesday to monitor what might be potential pain points.  Since a lot of participants here are now including hardware, OS and Office info I can get a feel towards what is happening.  If similar hardware configurations or hardware generations are not having issues, I am more confident.  The MVPs and others also know many more sources of information and share knowledge far more than my 2 eyes and time are able to gather.  I pay special attention to reports of desktop woes as laptops are much more complex and many times use proprietary hardware or OEM manufacturer software tools that it is much harder to discern trends.

On Point #2:  I will move forward.  I do not want to play the catchup game over months.  I prefer to age my patching over a few days and watch for the reported issues (as well as other anomalies), so I am not guessing which one may have been problematic.  Fortunately, I have yet to encounter a month where the various data points leave me without a path forward.  Even at the worst, of January-May 2018, I would do a backup, create a restore point, do an image, and then install the last of the potentially problematic patchs.  I was determined not to roll back in the April mess.  I prefer to proceed and possible fix rather than stall.  I have been lucky (knocking on wood).

I also prioritize the patches based upon the potential vulnerability, and whether the exploit is in the wild.  As a result, as a Group B person, the IE Rollup is first even though I do not use IE, followed by Office.

Point #3 is not a challenge for me as an individual, but it is a valid concern when dealing with both desktops and laptops.  At least I have standardized on Windows and Office versions (by choice and not design).

With all this though, I do recognize I have to move forward as some of my hardware is going to remain vulnerable (Spectre/Meltdown) and Windows 10 is not going to allow user control or privacy settings that matter.  That challenges for me is between Fruit or Birds, but until then, thanks for all you do, and it is helpful to individuals also.

Reply | Quote

Susan Bradley wrote:

…when I get to the point that I’m more scared about being unpatched than I am about patching that’s the point where I’m ready to patch.

That’s a great summary. Bravo, Susan.

And we know that knowledge eases fear of the unknown.

-Noel

2 users thanked author for this post.

Elly, Jan K.

Reply | Quote

What do I do.

I do not have the luxury of a canary.

I have two computers at home –
My own and
My wife’s Dell Inspiron 15R 5520 laptop.– January and February updates caused this computer to take 8 times longer to boot up than without the updates.

I addition to these I look after six computers where I do two days a week voluntry work. All computers have i5 processors (Haswell, Ivy bridge, or Sandy Bridge processors).
I built them, softwared them up and maintain them.
We also have a server running MS homesrver 2011 (now past EOL and does not get any updates from MS)

All updates done up to end of December 2017.

Windows Updates turned off —
Image of computers done with Macrium Reflect as of end of December.2017
Allow Reg key has also been installed by Avast antivirus on all machines.

Note — the six machines I maintain as a volunteer all have fixed IP addresses set in each machine so the whole network is effected by the March April, May, and now June Updates.
as per their explaination below.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

There is an issue with Windows and third-party software that is related to a missing file (oem<number>.inf).
Because of this issue, after you apply this update, the network interface controller will stop working.

1    To locate the network device, launch devmgmt.msc; it may appear under Other Devices.
2   To automatically rediscover the NIC and install drivers, select Scan for Hardware Changes from the Action menu.

a. Alternatively, install the drivers for the network device by right-clicking the device and choosing Update.
Then choose Search automatically for updated driver software or Browse my computer for driver software.”

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I do not see why I should have to risk breaking a complete fully functioning network and then have to rebuild NIC drivers again.
The disruption is not necessary without a whole lot more explaination from Microsoft as to why they are doing all of this.
I have the same brand of motherboard in each computer but different CPU,s and different Network chips.

As I understand it the update should FIX problems without creating more. It is a matter of getting it right before releasing the update.

To the poster with the 500 computers — My sympathy — I am upset enough with just 6 computer with no updates for 6 months.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mbhelwig

2 users thanked author for this post.

HiFlyer, Elly

Reply | Quote

So did you abandon KB4088878 / KB4088875 or did you approve them?

https://www.askwoody.com/patch-list-master/master-patch-list-2018-03/

Reply | Quote

I’m intrigued by this dichotomy.

On the one hand, there are pure technical amateurs like myself here, just trying to look after our own home (or perhaps even small business) machines, in some cases for family or friends, and benefiting enormously from the unpaid support provided here by Woody and his team of experts, and for which (I trust) we make voluntary contributions to keep the site running. On the other hand, there are apparently admins responsible for looking after hundreds or even thousands of business machines, who seemingly rely on the same unpaid support. If they’re responsible for that number of machines shouldn’t they either be IT experts themselves or else employing same?

In all honesty, I don’t see how a volunteer-run site like this one can be expected to provide advice to professional computer administrators, and I hope that it doesn’t gravitate away from its original purpose (as I understand it) namely to offer informal and unpaid advice on a largely volunteer and reader-led basis to a small and often amateur audience, without commitment or liability. I don’t believe that it was ever intended to provide more than that, but correct me if I’m wrong!

I don’t say that in a way that is intended to criticise those large-scale professionals who rely on the advice given here, it’s just that I’m not sure why or how this site could be expected to meet the requirements of that particular audience. There is also the professional indemnity aspect to consider – if a professional administrator relies on free advice from here to keep hundreds or thousands of business computers running, what happens if and when things go wrong? Surely those administrators would want a contractual arrangement with professional IT experts to fall back on in such circumstances rather than telling the shareholders that they based their patching procedures on the word of “some guy on the internet”? Maybe the site needs to make a prominent disclaimer or two!

On that basis, however, the site seems to me to be doing a great job as it is, and one that has only been enhanced by Susan Bradley’s arrival. I see her clearance of individual updates as being subject to the wider clearance of the month’s whole updating process as provided by the MS-DefCon ratings – and that’s fine by me, I don’t see any conflict or contradiction in that.

1 user thanked author for this post.

Noel Carboni

Reply | Quote

I’m IT in a mid-sized company.  I think I’m fairly typical when it comes to the position, namely I work multiple positions that would be full-fledged jobs in any larger organization.  I handle desktop updates, server updates, asset management, user support, some user training, programming, application support and application management with more on the way.  I don’t have days to go over the finer details of patches as they come out.  Usually, at least.  The Spectre and Meltdown kerfuffle ate up quite a bit of my time.

I use AskWoody as a quick guide of whether I should yay or nay the patch.  Depending on the issues that come out I’ll dive in in order to better understand what’s breaking and whether or not we can expect fallout if we do decide to go ahead.  Before I used AskWoody, I used to use sbsdiva (Susan Bradley’s old site) as well as a half-dozen other blogs to better understand where patch issues were being seen.  Back then I’d install all patches but the ones identified and then if something broke, I’d repeal the one that was causing the issue.  For actually determining what caused the error I’d normally search TechNet and Microsoft’s patch notes.

AskWoody has essentially replaced most of the blog crawling I used to do.  When errors occur, I use the links provided by Woody, but in general I still do my own research to determine whether or not I can justify installing a buggy patch.  The onus falls on me to verify the information given and make the risk assessment.  This isn’t something that the site can do for us.  Likewise I’m not paying the site for this service and I wouldn’t consider it liable for any of our security concerns.  There’s no contracts and no statement of intent of the site provide free services outside news and advice or opinion.  Any donations I make are just that and do not imply any form of liability on the site itself or the persons involved.

This is a long and drawn out way of saying I still have a job to do.  I can’t rely on AskWoody or any other site to do my job for me.  The buck stops with me.

5 users thanked author for this post.

Elly, columbia2011, Noel Carboni, Seff, wdburt1

Reply | Quote

Seff wrote:

On the one hand, there are pure technical amateurs like myself here
,,,
On the other hand, there are apparently admins

Yep, and I’m pretty sure Woody likes it that way.

IT folks were hitting this board long before Susan arrived, and undoubtedly many more followed her here. And that’s a good thing. A board with too many amateurs and too few alpha-geeks would be much less useful, helpful, informative, etc. So big thanks to the IT folks for the information they provide, the advice they give, and the questions they help answer here.

—
Note:
Before you ask, no, can’t explain the message from the guy with the unpatched 500+ VMs. In a world of 7+ billion people, there are bound to be outliers. But I wouldn’t make the mistake of trying to draw meaningful conclusions re the technical chops of most IT folks by extrapolating off a single data point.

Reply | Quote

To ME:
I’ll throw my hat somewhat into Susan’s ring…as she said, you need to have a pilot group (or two, three, or four). Anyone in the tech dept. gets patched first/second/third (depending on # of pilot groups). You should also have different update groups for hours to patch…space it out otherwise your WAN (and/or LAN if behind SCCM/WSUS) traffic is going to slow to a crawl.

At my old job (as an SCCM/SUS admin) we had desktop pilots and server pilots. Pilot desktop was IT support tiers 1-3, pilot server was test/lab environment. Pilots were pushed out 2 weeks after release, unless there were problems. Prod server/live desktop patches were pushed out 2 weeks later, or 1 week after pilot patches applied.

If you have 500 machines not patched since December, I highly suggest you stagger it, prod vs non-prod, and do pilots. And then keep that schedule. You should at this point probably jump to March or April at least. In my 2nd to last position (different company, no SCCM/SUS), we generally stayed 1 month behind on patches on both prod and test. You don’t necessarily have to jump straight to May 2018, but, March or April (whichever month they finally fixed all the stuff they broke with the Smeltdown patches).

I would also recommend patching prod systems case by case… for instance, leave app or SQL servers behind lesser non-critical stuff like file and print servers. I would patch DC’s first, assuming you have backups (but that’s another can of worms I hope you don’t have to worry about). Then do file/print servers, then app/SQL.

To Woody:
Even with Susan’s exceptional info, I still follow the DEFCON, as it’s easier to pay attention to (and I prefer the sledgehammer myself). The positions/jobs I’ve had that I didn’t follow the DEFCON (or was not the one who called the shots), the updates were generally done close to (if not at) the DEFCON rating change.

2 users thanked author for this post.

Seff

Reply | Quote

The first three months this year were a confusing mess. Ask Woody (the site) was helpful in supporting many diverse approaches to the administrator of many clients or end user during the confusion. So it is understandable to believe that holding at 12/2017 was endorsed here rather than recognized as an option. It wasn’t, but it was discussed in great detail and the volume of information could have lead to that interpretation. So thinking that way is possible.

But even the broadest tool in the garden shed, the sledgehammer MS-DEFCON has twice advised to move ahead for all instances that may have been on hold. The more recent publication in Computerworld was introduced on June 2nd on this site here and the Computerworld article followed by the next day. A similarly prominent all-clear was sounded for the April updates in early May.

It has been a mess, and difficulties have been documented elsewhere, referenced and discussed here. The suggestion is that Ask Woody was the source for the delay. But directions to move ahead with updates have been ignored. The Unofficial designation of volunteer regiment Group W has a legion or two. The details contributed from that contingent are helpful. But it is not an approach suggested by Woody’s articles, and the directions he gives does not endorse that system.

Patch Lady has extended a gracious invitation to @ME for assessment and reinterpretation of goals. The repair effort may have some bumps, but it will be some of the strongest advice available.

The mess was created elsewhere, and Microsoft was certainly a contributor with incomplete patches. Ask Woody (the site) is among the best resources for sorting the mess into appropriate piles. But the smelly work of dealing with the mess is on each of us. Unfortunate that @ME missed a couple of signals along the way, good that Woody (Da Boss) highlighted the outcome. It can be helpful to the dozens of others who are in the same situation, but have not shared.

2 users thanked author for this post.

Cybertooth

Reply | Quote

Simple. Patch only what you need. You don’t need others’ input on what to patch and when. Do your homework! READ THE KB ARTICLES.

Reply | Quote

The documentation is more sparse than ever (a whole page that says nothing more than “this resolves issues in Windows”)… There are fewer words than ever describing these things, and a larger percentage of the words than ever are just boilerplate…

The experience of others, when possible summarized by experts, is more valuable now than ever.

That being said… The decisions and responsibility are yours. And sometimes the testing.

For example, I scoured the internet for hard information on what the Spectre/Meltdown mitigations do to performance, and in the end NO ONE had measured it at the level I required. I had to actually do the updates myself to learn that it’s bad – and more importantly, just how bad for my uses.

I’ve mulled over why…

My needs and priorities are different than those of others. My hardware and choices of software are different than those of others. And I guess I am relatively unaffected by marketing hype, and I may be amongst only a few who consistently prefer a computer over a phone or tablet. I can’t believe it could just be because I care how well my computer systems run where others don’t.

Yet here we are, with Intel’s/Microsoft’s patches part of every set of cumulative update available for every system.

Even with the mitigations disabled (e.g. by InSpectre) up-to-date computer systems are just not as efficient as those running Windows patched to December’s level.

-Noel

5 users thanked author for this post.

AlphaCharlie, Elly, Bill C., JCCWsusser, geekdom

Reply | Quote

As Woody explains in the Home page, the DEFCOM is a blunt instrument. And one that, I believe, is strongly determined, these days, by when the inevitable torrent of both Windows 10  and Rollup issues starts to dwindle towards trickle status.

For my part, being Group B, I tend to follow closely the Patch Lady’s Master Patch List, although I am in charge of only one machine, my very own old Win 7 PC, and not of hundreds.

Because her entries on the patches du jour are binary: either they say that there are issues already reported, or “Not at this time”. Whether it is just for one, or for hundreds of machines, that there are some “issues reported”, whatever they might be, means, to even a non-techie like myself, that it’s better to “wait a bit, until this gets clarified, to see if it also applies to me.” And invariably, sooner or later, clarified it is. Issues reported so far: “Not at this time”? Well, if the “non-bad” news have remained unchanged for two or three weeks after the last Patch Tuesday, and I have not gathered, elsewhere, that there might be actual problems in spite of what it says there, then I go ahead, do my Group B “Geronimo!” stunt and patch away all those “Not at this time” updates — and all those with issues that do not apply to me.

And so far, after months of doing this,  I still walk among the living and so does, metaphorically speaking, my Windows 7 Pro, SP1, x64, Intel I-7 “sandy bridge”, 11-year old PC… A fool’s luck, no doubt.

 

Ex-Windows user (Win. 98, XP, 7); since mid-2017 using also macOS. Presently on Monterey 12.15 & sometimes running also Linux (Mint).

MacBook Pro circa mid-2015, 15" display, with 16GB 1600 GHz DDR3 RAM, 1 TB SSD, a Haswell architecture Intel CPU with 4 Cores and 8 Threads model i7-4870HQ @ 2.50GHz.
Intel Iris Pro GPU with Built-in Bus, VRAM 1.5 GB, Display 2880 x 1800 Retina, 24-Bit color.
macOS Monterey; browsers: Waterfox "Current", Vivaldi and (now and then) Chrome; security apps. Intego AV

Reply | Quote

I released January to April in one hit last month after patches seemed to be stable.

The main problem seems to be that machines then showed up all over the place on which patches they needed.

I use Security Only via WSUS, the quality rollups are denied.

However, when trying to patch for the last four months, some workstations would just leave out one or two of the months randomly. January always seemed to get included, but then February and March showed up as being needed by about 25-30% of the servers (randomly), and just about all of them wanted April.

Now, the machines are reporting they don’t need any more patches, even when 2018-02 and/or 2018-03 were never applied. This is on a combination of servers running 2008R2, 2012 and 2012R2.

Even when I then let the servers scan against Microsoft’s update servers the extra patches don’t show up as needed.

No matter where you go, there you are.

Reply | Quote

Some of the earlier security-only might’ve been fully superseded by the April/May updates as they fix some of the NIC issues that were introduced in March’s security-only and the individual fix released mid-March.

Basically they could break an April/May patched system.

Reply | Quote

“The Patch Lady approach is a scalpel. The MS-DEFCON approach is a sledge hammer.”

The patch that generated dismay was the Carnac patch issued three weeks after the initial patch with instructions to uninstall the initial patch, install the Carnac patch, then install the initial patch. Sledge hammer or scalpel would have made no difference. Blowtorch, anyone?

On permanent hiatus {with backup and coffee}
offline▸ Win10Pro 2004.19041.572 x64 i3-3220 RAM8GB HDD Firefox83.0b3 WindowsDefender
offline▸ Acer TravelMate P215-52 RAM8GB Win11Pro 22H2.22621.1265 x64 i5-10210U SSD Firefox106.0 MicrosoftDefender
online▸ Win11Pro 22H2.22621.1992 x64 i5-9400 RAM16GB HDD Firefox116.0b3 MicrosoftDefender

Reply | Quote