Botched CrowdStrike security update breaks Windows worldwide, causing BSOD and crashes

BrianInEngland said:

So CrowdStrike messed and are expecting Microsoft to fix it...

Click to expand...

CrowdStrike immediately took full responsibility and outlined a procedure to fix it, but it requires to tediously do a time consuming procedure on each and every PC in an organization. Microsoft took it upon themselves to help their customers and have put together an easier automated procedure to fix each computer. But no matter how you cut and slice it, each and every individual computer running CrowdStrike antivirus has to do the procedure.

Are there no computers affected that could be fixed remotely? Is a remote fix theoretically possible for a Windows computer that has a blue screen?

kelper said:

Are there no computers affected that could be fixed remotely? Is a remote fix theoretically possible for a Windows computer that has a blue screen?

Click to expand...

It would have to stay up long enough to be accessed remotely. As I understand it, the crash happens pretty shortly after boot.

Almighty1 said:

Stock market pricing and reality are two different things.... GameStop and AMC wouldn't exist all this time if that was the case. What you said is only one analysis....

"The CrowdStrike bug was caused by a NULL pointer dereference in C++, a common error in memory-unsafe languages. This type of bug can lead to crashes or hashtag#security vulnerabilities, especially in privileged code like system drivers.

Key points:
1. C/C++ allow direct memory manipulation, making pointer errors common
2. NULL pointer dereferences cause crashes in privileged code (e.g. drivers)
3. Modern tools like AddressSanitizer can help catch these bugs
4. Proper error checking and testing are critical for system-level code

Rust is seen as a safer alternative because:
1. It prevents NULL and dangling pointer issues by design
2. Memory safety is enforced at compile-time
3. It offers performance comparable to C/C++

Rust is the future, period. C and C++ are becoming outdated. Rust excels with bulletproof memory safety and cutting-edge features.

For rock-solid, secure systems code, hashtag#Rust is the best option. The smart money's already on Rust—it's time for everyone else to wake up and smell the oxidation.

hashtag#cybersecurity hashtag#infosec hashtag#software hashtag#defense

Link to the screenshot: LinkedIn "

View attachment 101881

Click to expand...

Don't really get - where you're going with this, as in: trying to defend their incompetence - or cut more branches under their feet (the latter makes sense based on what's stated, above but then... why? ).
As for Stock Market, truly wish i understood this better (generally speaking) - since it's the easiest way to get rich (to invest) - if you know what you're doing... After all, some lost their life savings - the same way (bad investments in stock market). That being said, checking the CrowdStrike's stock market - they lost -10% (while in the past lost more) - but for the time being (past days) - their stock line is neither going up nor down.... just frozen - almost a straight line. I presume - the shareholders are still in a state of confusion - unsure if it's best to sale stock or buy - further reflecting the current state of this company: Unclear.

What's clear (sort of) - the damages resulted from this mistake are close to 1 billion (more or less) - and someone has to pay - yet, CrowdStrike didn't show any responsibility in this regard. On the other hand... lawsuits don't work that way - not corporate lawsuits at least - where this incident... can be seen as an opportunity by law companies to milk this company for as much as they can (5 -10 billion $? who knows.... ). A big hit... but still not enough to kill a company of this size (never claimed that in the first place - that CS is done for). Tho again, i could be wrong (while more optimistic by nature) - and maybe it is enough...

As for that one analysis - what you posted from Zack Vorchies - pretty much supports the same thing. If anything, he goes into details explaining the actual process:



If anything, he's also addressing the Elephant in the room: Windows ecosystem being more of a house a glass (or castle of cards). Both the way Windows Drivers work but also C++ is rather outdated and fragile - even for past decade. I concur, Windows used in Hospital, Airports or other crucial fields of work is... baffling to me. But still, that doesn't excuse - CrowdStrike's long line of mistakes (both programing - but also common sense protocols handled by multiple departments) - which is why/how this happened. As mentioned in the other post - the file responsible for the crash is/was not the main issue - more like - the result of so many mistakes involving multiple departments.

As for that Rust referral (where again, i don't understand - why you used those details as a response to my previous post - it's not like i was promoting or praising C++), coincidentally - a dev from CrowdStrike had a different take on its bulletproof memory safety: How to Deal with Out-of-memory Conditions in Rust | CrowdStrike But i digress.

BrianInEngland said:

So CrowdStrike messed and are expecting Microsoft to fix it...

Click to expand...

No, never heard anything about Microsoft being asked to fix any of this. However, if there could be less impact to systems when a possible problem like this comes out, it would be a nice to have.

This was 100% the fault of Crowdstrike and customers using the crowdstrike product are the ones who had a bad Friday and weekend.

Surely Microsoft bears some of the blame since a single file, full of zeroes, was enough to prevent Windows from booting?

If CrowdStrike has 25% of the EDR market, who are the other major players?

EDR = endpoint detection and response

Just dropping this for a laugh

kelper said:

Surely Microsoft bears some of the blame since a single file, full of zeroes, was enough to prevent Windows from booting?

Click to expand...

Windows did boot successfully. Again this blue screen appeared shortly after the lock screen.

A remote fix is not possible fully because of the crash happening so frequently. However, some machines after restarting a ton of times slowly got the tiny update from crowdstrike and were sometimes able to install it. That's why on some systems for some people 15 restarts eventually worked.

But pretty much 90% of people have to manually either go into safe mode and delete the file or do the notepad trick from command prompt.

In other news, the building is fixed and I can finally relax.

kelper said:

If CrowdStrike has 25% of the EDR market, who are the other major players?

EDR = endpoint detection and response

Click to expand...

They have about 60% of the top fortune 500


Sentinal one, Fortinet are other brands that compete, Microsoft has defender endpoint etc.

kelper said:

Surely Microsoft bears some of the blame since a single file, full of zeroes, was enough to prevent Windows from booting?

Click to expand...

Not from where i am standing. The OS was fine and booting. Then, a kernel l mode driver from crowdstrike started up on a botched update from crowdstrike and caused the machine to bluescreen. This is a crowdstrike issue.

pseymour said:

As I understand it, the crash happens pretty shortly after boot.

Click to expand...

Like I earlier explained. In a true managed environment, every PC is configured to boot from PXE. This is done not only to be able to re-deploy the OS on every PC in such a way that doing so requires no manual intervention of course, but also to be able to stay in control of exactly what are the boot files that every PC receives, via TFTP.

Like I also earlier explained. On Windows, it is possible to create custom WinPE images for this exact purpose. All you really need to be able to learn how this all works is a DHCP server and a PXE server. A good example of a simple PXE server, that you can experiment with is iVentoy (not to be confused with Ventoy). Again, those who are completely oblivious to basic IT stuff like PXE should deserve no place in corporate IT management. For heaven's sake this is 2024. Not 1994...

"It is always recommended to test patches before installing them on all the systems in your network to ensure that there is no downtime due to faulty patches." Emphasis in bold is mine.

From: Patch Management | Automated Patch Management System - ManageEngine Endpoint Central

kelper said:

If CrowdStrike has 25% of the EDR market, who are the other major players?

EDR = endpoint detection and response

Click to expand...

Palo Alto Networks, SentinelOne, Microsoft, Trellix, Trend Micro and Sophos among others

A Guy

hdmi said:

Again, those who are completely oblivious to basic IT stuff like PXE should deserve no place in corporate IT management. For heaven's sake this is 2024. Not 1994...

Click to expand...

Would computers affected by the CrowdStrike bug be remotely bootable?

So I was curious and asked my higher ups why we don't use PXE in my town hall meeting and they told me we don't use PXE because it's incredibly insecure. It doesn't meet their "zero trust" intiantive. They apparently use intune on their side while I just get machines shipped with images on them that I deploy per use case via PDQ.

I asked them what makes it insecure and didn't get far, so now to do some internet research.....

hdmi said:

PC in such a way that doing so requires no manual intervention of course

Click to expand...

You still have to tell the machine to boot from pxe though?

andrew129260 said:

I asked them what makes it insecure and didn't get far, so now to do some internet research.....

Click to expand...

PXE is quite old and has no encryption or auth mechanism. If doing Zero Trust, you would use UEFI’s HTTP boot, ideally HTTPS.

pseymour said:

PXE is quite old and has no encryption or auth mechanism. If doing Zero Trust, you would use UEFI’s HTTP boot, ideally HTTPS.

Click to expand...

Thanks, that is what I thought but wasnt entirely sure. I should have used the new term. Oh well, the meeting is long over and I dont think its worth bringing it up again.

hdmi said:

"It is always recommended to test patches before installing them on all the systems in your network to ensure that there is no downtime due to faulty patches." Emphasis in bold is mine.

From: Patch Management | Automated Patch Management System - ManageEngine Endpoint Central

Click to expand...

For sure. I get that completely. I just never heard of someone testing smaller antivirus updates. Testing of new antivirus version/program upgrades for sure, but not definitions or other smaller updates. It doesn't seem that cloudstrike offered a way to delay this botched update either.

andrew129260 said:

I just never heard of someone testing smaller antivirus updates. Testing of new antivirus version/program upgrades for sure, but not definitions or other smaller updates. It doesn't seem that cloudstrike offered a way to delay this botched update either.

Click to expand...

I think that's spot on. I can't remember if it was linked in this thread or somewhere else, but someone mentioned we're about to have, or should have, a reckoning with EDR vendors after this one. We don't use CrowdStrike, but our product also does not seem to have a way to stage these types of updates. Hopefully everyone starts pushing their vendors for a system to do rings or whatever makes sense.

pseymour said:

I think that's spot on. I can't remember if it was linked in this thread or somewhere else, but someone mentioned we're about to have, or should have, a reckoning with EDR vendors after this one. We don't use CrowdStrike, but our product also does not seem to have a way to stage these types of updates. Hopefully everyone starts pushing their vendors for a system to do rings or whatever makes sense.

Click to expand...

Thanks. Yeah I mean, I am certainly not in that high of a position (thank god), but have friends who are. None of the smart people I know think there was anything that could be done to prevent this in this specific case. This was an extremely unlikely scenario. For the ones that do offer update management, they have options but it's limited for new version upgrades and the like, not these "smaller updates". Anything modern using antimalware solutions has cloud streaming updates, which are not possible to be managed.

Also, I have one friend who told me he cannot manage security updates when it comes to windows* or antimalware updates because the companies security insurance provider demands it be kept updated at all times. They don't allow slow rollout of security patches. They have to be allowed to update all all times to be complaint for the insurance protection. (They got hit with ransomware last year)

*To be clear he can delay all other windows patches, just not security ones.

Click to expand...

I think this problem is a lot more complex than just saying you need to have more rigorous patch management. There is so much more at play here that I don't think there are simple solutions to these problems. Things do need to change, for sure. But it also is about being realistic as well.