Server 2016 LTSC patches take for-e-ver. There are numerous reasons why – and not much you can do about it

Topic

New Reply

AskWoody reader (and Server expert) @alQamar has discovered a very disconcerting pattern when updating Server 2016 LTSC Core and LTSC GUI machines. An
[See the full post at: Server 2016 LTSC patches take for-e-ver. There are numerous reasons why – and not much you can do about it]

2 users thanked author for this post.

PhotM, Kirsty

Reply | Quote

Replies

Here’s Karl’s full original research:

Windows 2016 Server is 1607 – oh wait this includes 2016 LTSC clients too – d’oh! –
is a vista-like release too, and like it or not Microsoft does not care to service LTSC – by design AND by definition – unless there is something crucial broken or a security issue. The printer issue wasn’t one of these and the next thing which is Update speed issues – isn’t either. There are tons of reactions about Server 2016 and Client being slow updating. While most reliability issues have been fixed by today – update speed has not, not even with the latest SSU 02-2019.

As promised to Woody and a clique of Windows warriors I’ve made my promise true and made amateurish videos that display the issue uncovered:

Server GUI: The first video is 46 mins long – and you can skip forward from minute 5

Server Core: The second video is 25 long and you can skip forward from minute 4.

So why would I record a lengthy video that possible extend any modern attention span and
recommend to skip forward? Because of mind to not wasting your time and plain boringness to watch while Server 2016 is still “downloading” and patching!

You get the figures right: Server 2019 (1809) needs up to a tenth of the time Server 2016 (1607) needs for patching a single batch of monthly update. In a short: It is a pain.

*remarks: Due to a bug in 1607 download and install phase are wrong (fixed in Server SAC / Client 1703 and later), the DL is finished but it changes to install phase later while tiworker is already running. A 450 mbit line is not the bottleneck to DL patches from WU. So, install time cannot measured well and needs to be seen additive.

Please note that I had hard times to stop time with the GUI and core server, so please refrain arguing about my measurements being incorrect – I’ve corrected these from notepad later by re-watching the records – or discussing the methodology of patching them manually lacks automation skills – intention of the videos is solely to bring up the issue AS IS and which is reportingly known to Microsoft via technet and uservoice for at least since 2017 and nothing happens at all!

This does not only show some sorts of ignorance in my humble opinion, but also that uservoice and feedback hub simply does not catch issues enough rather than dealing fancy design improvements. Product design issues are not debatable it seems. Refer 1809 release and other disasters.

I’ve come to the point to stop recommending Server 2016 altogether for our customers and use this as one single of many points for going to 2019 alone – there are many more benefits though – whilst 2019 licenses are more expensive, too.

Lastly it shows that LTSC -again by design- does not work as intended and MS does want customers upgrade to the latest release whenever possible, no matter if server or clients. Except you are very forgivable on issues.

Under these circumstances, buying licenses without Software Assurance (roughly +50% the license price for 2 years) becomes no longer an optional decision as a side-effect. Higher prices + selling SA is a double win, but mainly for Microsoft.

There have been publications of Günter Born and a survey, nothing of that was strong enough to get Microsoft done to update their servicing stack issue in Windows 1607.

One might say, ok but who does patch servers manually, get this orchestrated. I agree again but orchestration does not come for free and it depends the size and budget of the company if this would make sense for 5-12 servers, too.

What do think? Let me know in the comments below. Thanks for reading and spreading the word, until MSFT is going to fix this – before end of main support in 2021, since no such problems will be fixed but only security issues in extended support by design.

5 users thanked author for this post.

warrenrumak, NetDef, abbodi86, lawrenceB, PhotM

Reply | Quote

Thank you for this. I have a few Server 2016 GUI VMs, and they’re all brutally slow to update.

Unfortunately, I bought a bunch of 2016 licenses about 6 months ago, opting out of an overpriced Campus server agreement, so 2019 isn’t really an option for me.

I assumed they would eventually resolve these issues (the MS forum threads are full of people with similar complaints.) But 2016 has been out for over two years, and I’m starting to wonder.

 

1 user thanked author for this post.

alQamar

Reply | Quote

another reference in technet:

https://social.technet.microsoft.com/Forums/en-US/250914a1-b013-434a-baa0-639e31a839dd/whats-with-the-really-slow-windows-updates-on-2016?forum=ws2016

1 user thanked author for this post.

alQamar

Reply | Quote

I see similar behavior on the one Server 2016 I have (which is a ShoreTel Director server appliance).  I figured it was just underpowered but having read this, it seems that may just be how it is.

1 user thanked author for this post.

alQamar

Reply | Quote

woody wrote:

Here’s Karl’s full original research: Windows 2016 Server is 1607 – oh wait this includes 2016 LTSC clients too – d’oh! – is a vista-like release too, and like it or not Microsoft does not care to service LTSC – by design AND by definition – unless there is something crucial broken or a security issue. The printer issue wasn’t one of these and the next thing which is Update speed issues – isn’t either. There are tons of reactions about Server 2016 and Client being slow updating.[…]

What do think? Let me know in the comments below. Thanks for reading and spreading the word, until MSFT is going to fix this – before end of main support in 2021, since no such problems will be fixed but only security issues in extended support by design.

I have been a Microsoft Partner for over 25 years. What I’ve seen is a company that had reps that provided expert help and worked hard to resolve issues like this, to a company that today puts on a happy face, but really seems to not care anymore about satisfying customers. With Windows no longer a core focus at Microsoft, I expect this lack of customer satisfaction to continue. Hopefully, it won’t get worse.

GreatAndPowerfulTech

1 user thanked author for this post.

NetDef

Reply | Quote

Is this downloading from Microsoft, or locally from a WSUS server?

I’ve given up doing anything from Microsoft’s servers, for my domain-joined servers, I use WSUS, for the others I use WSUS Offline.

No matter where you go, there you are.

1 user thanked author for this post.

alQamar

Reply | Quote

If you have a volume license, you can clean install Windows Server build 1803 or back to 1709, but there are some caveats. You cannot activate them on your Server 2016 non-volume license keys. Nor are they sold at retail or on a pre-built retail OEM server.

They’re not technically Server 2016 or 2019 (which is build 1809 . . . confused yet? )

Those two intermediate builds are Core and Nano only: no GUI.

If you need/want the Essentials Role, you need 1607 or even back to 2012 R2.

The two GUI builds based on the same kernel as Win10 are 1607 (Windows Server 2016) and 1809 (Server 2019) – and even then there are some odd offerings that don’t quite map cleanly to those builds either.

Having made that clear as mud, it’s very true that all the builds after 1607 do updates quite a bit faster . . .

~ Group "Weekend" ~

1 user thanked author for this post.

alQamar

Reply | Quote

1607, even in its Windows 10 client form, always had major issues with updates. We had a large number of 1607 machines just hang and trash the disk while downloading updates. On a HDD-based system, it was often enough to grind the machine to a halt. Despite logging several support cases with Microsoft, they treated it as an isolated issue and just installed the offending update manually. We eventually discovered that upgrading away from 1607 was the only real solution. Obviously not an easy option for Server 2016 though.

1 user thanked author for this post.

alQamar

Reply | Quote

Comparing 1607 with 1809 is not reasonable
current cumulative update of 1607 is huge and replace more than 70% of inbox components, while 1809 is still fresh
not to mention that 1809 has a new format for cumulative updates called PSFX, which reduce the payload size and probably the patching time

Servicing Stack Update cannot solve the design flows in the stack, they only fix the issues and try to handle new cumulative updates correctly

4 users thanked author for this post.

NetDef, alQamar, woody, PKCano

Reply | Quote

We have roughly 300 Win2016’s running, mostly with GUI. I had noticed slow updating on some of them (the install in Software Center would time-out after 60mins and report a failure), but in my experience it’s not all our Win2016s that are slow.

But we’re using SCCM to patch our cattle, so I’m not right on top of them while they’re crunching bits.

I haven’t compared slow vs fast till now. I just assumed the slower ones had less CPU resources available during the update process. But worth some more thorough comparison in future!

1 user thanked author for this post.

NetDef

Reply | Quote

abbodi86 wrote:

Comparing 1607 with 1809 is not reasonable

I agree the 1809 are much smaller by design (express updates), and in GUI you might even see a linear dependency of update package size and time.

we all can review the GUI based update package sizes here, at a glance: https://www.computerbase.de/downloads/betriebssysteme/windows-10-kumulatives-update/

Comparing them makes still sense for me, as neither any later Server “2016” such as Server 1709 Core or 1803 Core suffers from the same issues – whilst having the same big batches to work on and will be much faster than 1607 Core.

And neither does this apply to any of the Servers before, even though they all share cumulative updates by now.

To make it clear: you can wait for patching one Server 2016 1607, while you are patching 20 or more 2012 R2 server or even 2019 while waiting the one Server 2016 to complete. And that is not a lack of ressources in CPU and storage. It just happens.

One Patchday we started a 2016 machine at 5 pm it was finished after all others at 11 pm, installed 2 CUs during this time, and that’s what other report in uservoice and technet, too.

Freeco’s comment is interesting and it does actually match that some users say they do not see any speed issues here.

1 user thanked author for this post.

NetDef

Reply | Quote

“Is this downloading from Microsoft, or locally from a WSUS server?

 

Actually it does not matter. The outcome is same for WSUS and WU (haven’t tried WuFB / peer2peer)

The confusing part is, as stated in the remarks, the download time appear to be much longer given the table. This is not the case.

 

The download is fast with the latest CU.
The problem is that 1607 reports to “Download” in GUI and Core while it is already installing, so most of the time it sits there installing (tiworker.exe).

Add the DL time to the installation time in the table above. my line wasn’t busy for long DL 1.5 GB.

Reply | Quote

Just download the update from the Microsoft Catalogue, extract the cab file and use DISM to install the cab.  It improves the speed significantly.

Reply | Quote

This does work, but occasionally you might run into a major gotcha dependency that’s not well documented. Recently for example there was a tiny utility called PCIClearStaleCache.exe that was invoked by WU, but not the Catalog installer (although it was offered as part of the download on a separate link.

Failing to run that prior to the patch resulted in odd results.

~ Group "Weekend" ~

Reply | Quote

If you downloaded both and put them in the same location, the update executed the .exe in the process. You could see the cmd prompt flash (if you had quick eyes).

1 user thanked author for this post.

NetDef

Reply | Quote

Great finding. So if a manual Dism install is faster than the automated process wouldn’t this reinforce the point it’s not just the bigger size of updates but a defect in the automated process???

1 user thanked author for this post.

alQamar

Reply | Quote

Karl’s empirical data confirms, what >430 users voted on a survey, I’ve set up last year

Survey: Slow Windows Server 2016 Update installs?

Here is an article I’ve created from the data I received from Karl

Windows Server 2016: Empirical proof of slow Update installs

Ex Microsoft Windows (Insider) MVP, Microsoft Answers Community Moderator, Blogger, Book author

https://www.borncity.com/win/

4 users thanked author for this post.

alQamar, Kirsty, woody, NetDef

Reply | Quote

Hi everyone,

I cannot agree that manual download / installation / extraction of the MSU files will speed up things (much).

I’ve had hard times <span style=”display: inline !important; float: none; background-color: #ffffff; color: #333333; font-family: Georgia,’Times New Roman’,’Bitstream Charter’,Times,serif; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px;”>this morning</span> to find out in my lab. Definitely though, I defend, that the update size and OS age is not the reason for a slow patching LTSC 1607.
It is in the code.

Watch yourself: https://1drv.ms/v/s!ApTx3d3fhinPgpk-2l8Vku1C26Pz7A

Patching Server 2016 LTSC Core 1607 vs Server 2016 SAC Core 1803 + 2 installations from scratch (each about 3 minutes)

Server 2016 LTSC Core 1607
Size 1393,5 MB
Search Updates > 13 sec
“downloading” Updates > aborted after 28 minutes

Server 2016 SAC Core 1803
Size 835,8 MB + SSU 1,2 MB + Delta Update 287,6
Search Updates > 19 sec
Downloading Updates > 2 min
Installing Updates > 2 min 45 sec
Reboot / Applying Updates / logon > 48 sec (typo password measured 57)
Total Patch Times > 6 min 56 sec

 

Transscript:
Server 2016 LTSC shows the usual behaviour to report download while already installing the update. The status is not consistent with the actual process. This is valid for GUI and core and was fixed later in W10 1703 / Server 2016 SAC 1709
As we see now the process is broken by design, not by update sizes as abbodi86 stated.
The Server 2016 SAC 1803 has quite similar update speed as Server 2019
Despite the package size of 2019 is even smaller.
I don’t expect 2016 1607 Core to finish before minute 25 as last time.

If I am able to install a server in a time of 3 minutes the system cannot be powerless enough to patch anything within a reasonable time.
Just for fun we could install a second 2016 server from scratch to make the sillyness of this bug obvious.

I could literally setup 15 servers + patch them – in a timeframe where one of these 2016 1607 would be completed patching, or more depending if GUI or not.

Showing this a complete clean installation should remove all doubts the issue is by design, and MSFT has fixed it in later code.
Soon™ finished
you see the new 2016 server 1607 suffers even more issues. longer update search etc. so the 02-2019 and older SSU improved it – except the installation time and wrong display of “download” phase

ok at least there seems to be a relation of SSU and download if no older SSU was installed – interesting enough

KB3192137 – mysterious…
https://social.technet.microsoft.com/Forums/Azure/en-US/28b60975-c6bf-4986-8d80-7be953597a5a/what-is-update-kb3192137?forum=ws2016

nevertheless even the new installed 2016 has the same installation speed, despite the download issue appears to be fixed.
seems the 2016 LTSC is crashed…
0 percent CPU makes me worried
But I think you get the point, the new installed 1607 isnt that fast either.
I will stop it here. not worth the time.

 

Reply | Quote

anonymous wrote:

Great finding. So if a manual Dism install is faster than the automated process wouldn’t this reinforce the point it’s not just the bigger size of updates but a defect in the automated process???

it does not. I put some time into this. it is faster but not as fast as it should / could be – depending the hardware – you can probably save your time to script this into PS.

Reply | Quote

If anyone is questioning why I put so much time and passion into this

1. Server 2016 is the same crapware as Server 2008 and 2012 (without R2).

The rule that every second OS release is a better one perfectly lines up with Windows 10 AND Servers (except 1511, that was merely a hotfix to avoid waiting for 1607).

The other point is, that I want to proove that LTSC is not something you can rely on for Servers and Clients alike.

2. Getting the latest release can be reasonable at a certain time, but it seems it is no longer optional – not even for LTSC.
This is all possible with inplace upgrades this is easily done from 2008 to 2019. But you need Software Assurance to do so / or another license agreement (Providers, IaaS etc)

The famous printer control bug in LTSC 2015 1507 gives it another drift.

3. For Servers, however, there are so many dependencies – VMware Level, Citrix, Backup, etc. Microsoft cannot expect anyone to upgrade to Server 2019 just to have a performant and more reliable OS – it is NOT a Client.

4. 2016 is still in main support. PERIOD.
By Microsofts support definition it has to be patched, if there are functional issues – this is valid – except for LTSC.

It deserves a backport of whatever MSFT has incorporated in 1703 or later to solve the issue

1 user thanked author for this post.

NetDef

Reply | Quote

I waited a year to upgrade to Server 2016 and still experienced severe problems with Hyper-V backup.  Veeam implemented a very nice feature making use of ReFS block cloning and that exposed terrible performance issues in the ReFS driver that it took a year or so to get fixed.  Microsoft almost didn’t back-port the fixes to Server 2016.

I agree, Server 2016 is like a Vista.

Reply | Quote

Fully agree with alQamar (and a gracious tip of my hat to your efforts) as my small setup involves a mixture of 2008 R2, 2012 R2 and the infamous 2016, and yes 2016 is a royal time waster installing updates with the same update on different 2016 machines sometimes taking closer to an hour, where the rest taking less than 15 minutes (though that’s a rarity these days).  My solution, trigger the updates the night before, log back in early morning, trigger the reboot, and then make my way into work (so far it’s been reliable).
Again, to underline alQamar’s results, the kicker.  When I bought our 2016 server (as it was still cheaper than MS365 over 5 years) the installer tried to convince me to go with 2012, and they were right (as proven by the reliability) and wrong (as I’m looking to maintain our investment for as long as we can and 2012 would have cut the useful life significantly).
Realizing we each have different needs, and Microsoft makes my / our life h****** at times (Office 2019 anyone) it is becoming more difficult to keep our hard earned funds out of Microsoft’s pockets then it needs to be.

Edit to remove HTML. Please use the “Text” tab in the entry box when you copy/paste.

IT Manager Geek

2 users thanked author for this post.

alQamar, NetDef

Reply | Quote

We support several deployments both on hardware and in Hyper-V of

Server 2008 R2 *in migration planning by Jan 2020*
SBS 2011 (think Essentials Role but on 2008 R2)
Server 2012 R2 (many with Essentials Role)
Server 2016 (also many with Essentials Role)
A few NEW Server 2019s.

And a mix of application only servers and AD/DC servers.

One of the things I’ve noticed over the years is that all of the above with AD always take considerably longer to patch. And I agree that Server 2016 takes far longer to patch than any of the others on that list with or without the AD role (but even longer with AD.)

But, so far at least – our Server 2016’s while slow to patch – have run like champs since deployment. And their performance for everything else is just great when they are fed decent RAM and SSD arrays on a moderately fast CPU.

We’ve learned to live with the patching speeds. Although it would be nice to get improvement, my understanding is it would take a feature release to fix – which I am 99.9% certain is not in the cards.

~ Group "Weekend" ~

1 user thanked author for this post.

alQamar

Reply | Quote

KB4487026 via sccm timed out (max run time 360 min) granted Dev and Test environments are on slow drives.

I gotta feeling patching night will take a couple more hours for prod

 

Reply | Quote