Botched CrowdStrike security update breaks Windows worldwide, causing BSOD and crashes

A Guy

A Guy said:

View attachment 102197

A Guy

Click to expand...

it would seem that his old habits are still well and truly as bad as before.
i do remember the 'McAfee Malarky' but i was on Ubuntu Linux at the time, and smiled ..

mingle said:

Blown out of all proportion - I've been running 7 different systems, using all sorts of services and didn't notice ANY issues.

Still, let's not deprive the media of their weekly fix of sensationalism, eh?

Click to expand...

A Guy

pparks1 said:

We had nearly 1000 servers down and thousands of employee workstations to fix from this issue. It wasn't sensationalism at all

Click to expand...

That's a best case scenario. There's plenty of companies - who are rather cheap with their IT Department (2 - 5 guys handling hundreds of systems). Poor guys, kept booting systems in safe mode and deleting the CS file - manually - as if working on a production line.

A Guy said:

A Guy

Click to expand...

He's saying the same thing - casting same blames in both of his videos:


Tho, i don't like - how he's treating Apple at face value - as a multi billion $ company - who cares about its ecosystem and improves it in such way - that every 3rd party joining the OSX Club have to play by their rules. As was admittedly the case even with CrowdStrike - whom seeing they can have it their way - they adapted - and build their software around OSX's ecosystem following Apple's rules . Microsoft on the other hand (Crowd-strike too) - are treated more like indie companies - who lack both the resources and skilled programmers to improve the OS/software. He admits that Microsoft tried something in this regard by working on a security API - which could have worked with CrowdStrike's software - as a fully integrated part of Windows (not like now - where it has ring 0 priority), but... the non-technical officials from EU - manged to easily rule it out - just for the sake of giving every 3rd party app a fair chance - at crashing Windows.

Thus, this indie company called Microsoft with 3 students programing Windows from Bill Gate's basement - decided to just let everyone in... to take a dump in the middle of their house (Windows ecosystem), as long as it passes the WHQL (Windows Hardware Quality Labs) release signature (or digitally signed driver package), tho... rather pointless/useless for CrowdStrike "since the certified files were dynamic (can still be subjected to changes - even after passing the WHQL)" - as was supposedly the case with the file which caused the BSOD loop. If anything, this incident is actually a good thing for the masses - since it revealed serious issues with the coding of the software - while CS devs were actively ignoring by following a meme mentality ("if my chance/luck still works - avoid messing with it"), but also issues with other departments - quality control & testing in particular. Maybe even the aggressive updates policies enforced on Channel File - more focused on dealing with zero-day vulnerabilities - by sacrificing/risking the system stability (2 decades of enterprise policies meant to avoid this type of issues - thrown out the Windows by CrowdStrike and others alike). Same can be said - about the elephant in the room (fragile state of Windows). And thus, the outcome...


Yet, realistically and objectively speaking (taking empirical statistics into account) - Microsoft, same as Apple - net-worth wise.... is not just a multi-billion $ corporation, but trillions... If Apple afforded to show EU the middle finger "while in the wrong" - Microsoft afforded the same "for the right reasons". And FFS... it's about time they rework Windows from the ground-up (if necessary) - instead of keep adding more rubbish on top of an a fragile and outdated ecosystem. Even Gaming companies - are taking dumps on Windows security this days, as in: it became common for quite a lot of them to implement "Kernel Level Anti-cheat" with the their games (which functionality wise, works very similar to CS Falcon Sensor - but instead of scanning for malicious content - those scan for cheats/bots/scripts). Again, same games - running in OSX "can't do that - because Apple doesn't allow it". As for CrowdStrike's quick rise to financial success - proved to be yet ANOTHER Cybersecurity Company who bite off more than it can chew (like Norton, McAfee - kinda Avast...) - a company which managed to climb from 35 Billion $ in 1 July 2023 to 95 Billion $ in 1 July 2024, tho... more focused on enjoying the ride and maybe investing more in marketing than software development, testing and maintenance.. And thus, while having this Titanic like experience - in less than 30 Days their stocks market cap dropped to 65 billion $ - still "frozen" at this value. The CEO alone lost 600 mil $ (never had 1 so i have no idea what this means). While more losses close to 5 Billions are expected, but hey - as compensation for your trouble CrowdStrike to vendors: Sorry for the global tech outage. Here’s a $10 Uber Eats voucher



Sure, bread and circuses.... but at least it can bring some laughter on some faces (we all know Microsoft and all the other big boys - would never do that if guilty of same thing).

@pparks1 1000 servers !!!!! that's HUGE !!!!!!

Perhaps if companies had deployed more of this type of strategy then the effects would have been far less.

Have physical servers using say LINUX or UNIX type OS's hosting Windows VIRTUAL Servers.

Then the update only needs to be applied to the Virtual servers -- if proper Virtual Desktop Infrastructure is deployed at the users end then no update at all of this nature needs to be pushed out to the end users Windows machines (that saves loads of 8 - 10million end user machines from having to be regularly updated).

Now if the Windows Virtual server gets a problem -- note Windows does load initially so any logs based on Windows loading would be incorrect in this case, however if the Windows virtual server crashes then it's not reachable. In this case Linux / Unix can automatically restore the previous (working) image(s) of the Windows virtual server(s) and the end users should then be able to logon as usual.

In any case robust testing by the AV supplier should of course have taken place but I'm not sure why the strategy I've outlined here which seems to offer a lot more protection against failing updates etc isn't more used. Virtual machines on decent Enterprise systems can function really efficiently these days especially in those server farms using blade servers etc.

Even if the Unix / Linux system gets attacked (it's much harder than Windows of course but no OS is 100% impregnable) it's much easier to recover the server(s) as there's usually staff on hand in those places to do that -- with the current system all those end users computers have to be manually fixed -- and these days most end users don't have the skill to do it themselves and typical even big companies have often outsourced their entire I.T depts so getting the staff to manually fix these machines takes in some cases days to obtain.

There's a time and place for Virtual servers and IMHO this seems one obvious place for their use.

Cheers
jimbo

neves said:

as a fully integrated part of Windows (not like now - where it has ring 0 priority), but... the non-technical officials from EU - manged to easily rule it out - just for the sake of giving every 3rd party app a fair chance - at crashing Windows.

Click to expand...

McAfee as a fully integrated part of Windows? That would be even worse than buying an Apple so, if what you say about EU officials is true, if there's anything that makes me proud of being a EU citizen, these officials will probably be it. Speaking of the word citizen... have you watched the 2014 documentary film Citizenfour? I recommend it a lot...

neves said:

If Apple afforded to show EU the middle finger "while in the wrong" - Microsoft afforded the same "for the right reasons".

Click to expand...

Yuppers, these people are very rude. Showing EU the middle finger and all that... Manners maketh man.

neves said:

And FFS... it's about time they rework Windows from the ground-up (if necessary) - instead of keep adding more rubbish on top of an a fragile and outdated ecosystem.

Click to expand...

I was going to. But then I got lazy.

neves said:

, but hey - as compensation for your trouble CrowdStrike to vendors: Sorry for the global tech outage. Here’s a $10 Uber Eats voucher

Click to expand...

This makes me feel hungry so, I am going to make my french fries now. See you all in a bit, and, stay classy, internet!

jimbo45 said:

@pparks1 1000 servers !!!!! that's HUGE !!!!!!

Have physical servers using say LINUX or UNIX type OS's hosting Windows VIRTUAL Servers.

Click to expand...

We do run virtual servers for nearly everything. We run on VMWare VSphere on blade servers.

jimbo45 said:

if proper Virtual Desktop Infrastructure is deployed at the users end then no update at all of this nature needs to be pushed out to the end users Windows machines (that saves loads of 8 - 10million end user machines from having to be regularly updated).

Click to expand...

A virtualized desktop environment for our employees is not feasable for our workforce. Employees have Windows or Mac laptops.

jimbo45 said:

There's a time and place for Virtual servers and IMHO this seems one obvious place for their use.

Click to expand...

Yes, enterprises have been running on virtual servers for many years. New things are shifting to the cloud which don't require maintaining infrastructure and servers...just infrastructure code which can self heal. Other things are now containerized and running on platforms like Kubernetes and don't run full OS's.

Honestly, we are at the point where full virtualized servers are "old school", and we are moving away from them.

I'm no programmer but is it not possible to make a "smart" boot loader which automatically, unloads and skips bad drivers and continues with thr boot sequence instead of crashing?

ibay770 said:

I'm no programmer but is it not possible to make a "smart" boot loader which automatically, unloads and skips bad drivers and continues with thr boot sequence instead of crashing?

Click to expand...

Anything possible for a system to do, is also possible for malware. If you have something that can not load the driver of an AV and skip it, malware could use this technique as well.

It's risky. But yes, things could be designed in such a way that if something fails a certain amount of times, something could take over and ensure windows boots without the troublesome file.

The trouble is finding that balance.

ibay770 said:

I'm no programmer but is it not possible to make a "smart" boot loader which automatically, unloads and skips bad drivers and continues with thr boot sequence instead of crashing?

Click to expand...

This is way more complex. Most drivers/apps and such run in user space and don't take the system down when they crash out. However, because of the nature of the way these security apps work, they HAVE to run in kernel space. And if you watch that video above from Daves Garage he does a great job explaining that when something crashes in kernel space, BY DESIGN, the system is immediately halted (blue screens). The risk is that allowing something in the kernel that has crashed to allow the kernel to keep running could make things even worse.

pparks1 said:

This is way more complex. Most drivers/apps and such run in user space and don't take the system down when they crash out. However, because of the nature of the way these security apps work, they HAVE to run in kernel space. And if you watch that video above from Daves Garage he does a great job explaining that when something crashes in kernel space, BY DESIGN, the system is immediately halted (blue screens). The risk is that allowing something in the kernel that has crashed to allow the kernel to keep running could make things even worse.

Click to expand...

much better said.

andrew129260 said:

much better said.

Click to expand...

Another reason never to run repairs or virus removal stuff on a booted up faulty machine --- would you as a pilot willingly take off in a defective plane and get it fixed in the air !!!!!.

Cheers
jimbo

andrew129260 said:

much better said.

Click to expand...

I see @Brink already posted the link that splains how they're going to fix it: VBS enclaves. The VBS conclaves are still only just a rumor for now.

A Guy

estimated cost $5.4 Billion .. thats one expensive file error ..

XxXxX said:

estimated cost $5.4 Billion .. thats one expensive file error ..

Click to expand...

It's just a few zeros.

hdmi said:

It's just a few zeros.

Click to expand...

Like the botched update file! Zing!

pseymour said:

Like the botched update file! Zing!

Click to expand...

Exactly. lol Tech Analysis: Channel File May Contain Null Bytes | CrowdStrike

In this blog post, we examine the recent CrowdStrike outage and provide a technical overview of the root cause.