Botched CrowdStrike security update breaks Windows worldwide, causing BSOD and crashes

NormanF · Jul 20, 2024

I'm running Sophos MDR and there are no issues with sensor deployment that feed Microsoft Defender data into the lake cloud.

XxXxX · Jul 20, 2024

all this does is remind me how the world has come to rely on tech so much.
an update breaks systems worldwide and the world goes into meltdown.

oh well something to look forward too ..

kelper · Jul 20, 2024

Could virtualization have avoided this or, at least, made recovery easier? Is it true that Windows 11 machines can't be affected?

hdmi · Jul 20, 2024

andrew129260 said:
2024 CrowdStrike-related IT outages - Wikipedia

en.wikipedia.org

Now I can finally sleep.

"United States President Joe Biden was in touch with CrowdStrike to offer any assistance if necessary."

TairikuOkami · Jul 20, 2024

This is a server side issue. Smart admins do not install latest updates ASAP, because of this. NP here. :yawn:

neves · Jul 20, 2024

Almighty1 said:
They might or the company might be worth investing in as it shows how much everyone depended on them.

If that's the case - that's just "Marketing (deceit/manipulation)" at this point.

Cause on the technical side of things - this mistake proved to be the result of a kindergarten level of programing incompetence - revealing multiple serious issues with the state of the software and its programing. As in, we know now that the file/update responsible for this crash - was full of null characters:

It's what caused the crash, but that not the big concern - more like... WHY!? And... HOW (did it got to this point)?! What level of programing, testing and management did this "indie company" actually have - while handling a worldwide critical task?! Or is CrowdStrike handled by a student with a GitHub account - experimenting with some personal project (a bad student too - cause there's tons of programmers on GitHub - who wouldn't make this mistake simply by following basic protocols). Seriously...

1 - How was a null file pushed for updates - without being tested and signed (the most basic protocol - which every company with commercial products tends to have in place)? After all - this is a critical update - for a product used at an enterprise/public level (even by airports and hospitals) - and yet - somehow managed to miss both marks.

2 - Again, this being a critical update for a critical product - a second verification should take place on the client side as well. And yet, this verification - is either missing from the software or someone kid introduced a coding statement like: - if Null, Skip

3 - Why did the Kernel Module Crash while reading this file?!?!?!?!?!?!?! That shouldn't be possible in the first place (treating a null file as if it contains readable data to work with). That's quite literally - like Siri telling you to Turn Right off a cliff... and Crash - while you can see clearly on the GPS that there's no road there - but the software is so badly programed - that tries to forcefully read that data (and Crash as a result).

But hey, maybe their marketing team can sell sand in the desert. After all, at this point in time - it's less about the quality of a product - and more about the marketing/deception skill behind it.

TM21 · Jul 20, 2024

Yesterday, I was on-line with a major file server, possibly in Canada, downloading a ~1 TB file, when I got the BSOD.
My PC's do NOT BSOD, so that really threw me for a loop!!!
I immediately shut down my PC, changed to a different SSD, with Windows 11/Pro/64 on it, reconnected to the Server and finally got my download file. No more BSOD's.

Today, I checked my SSD's for the \CloudStrike\ directory, and I don't have it!
So, something must have downloaded from the File Server that caused my PC to BSOD.
That's the best explanation that I've got.

May it never happen again!

TM

kelper · Jul 20, 2024

@TechnoMage2021

Which error code did you get?

Ghot · Jul 20, 2024

TechnoMage2021 said:
downloading a ~1 TB file

That's a really big file.
That's like... 1000GB. (Godzilla-bytes)

I think you're just supposed to download the files ON the server, not the entire server.

Fabler2 · Jul 20, 2024

I wonder when the court cases will begin?

dlross · Jul 20, 2024

Fabler2 said:
I wonder when the court cases will begin?

Yesterday

pseymour · Jul 20, 2024

@andrew129260 I can empathize, man. My mother worked in radio and TV most of her career (not IT), and it is how you describe. She was not even allowed to change her wallpaper lest it might “consume extra resources.”

I’ll be continuing to pour one out for the IT homies this weekend, and pray for the sanity of those dealing with this.

Lawnmower74 · Jul 20, 2024

I have had a update KB5035942 in other updates is this anything to do with this problem. I have tried to find information about it but I cannot get onto windows site just shows blank screen.

andrew129260 · Jul 20, 2024

TairikuOkami said:
This is a server side issue. Smart admins do not install latest updates ASAP, because of this. NP here.

So you control when your crowdstrike antivirus updates? This was not a windows update.

kelper said:
Could virtualization have avoided this or, at least, made recovery easier? Is it true that Windows 11 machines can't be affected?

Windows 11 was affected. The virtual machines were hit with this as well, but thanks to snapshots could be reverted quite easily. Then when booted received the new update from cloud strike fixing the issue.

TechnoMage2021 said:
Today, I checked my SSD's for the \CloudStrike\ directory, and I don't have it!

Of course you don't, because you don't use crowd strike antivirus.

Lawnmower74 said:
I have had a update KB5035942 in other updates is this anything to do with this problem. I have tried to find information about it but I cannot get onto windows site just shows blank screen.

This wasn't a Windows update issue

I can empathize, man. My mother worked in radio and TV most of her career (not IT), and it is how you describe. She was not even allowed to change her wallpaper lest it might “consume extra resources.”

I’ll be continuing to pour one out for the IT homies this weekend, and pray for the sanity of those dealing with this.

Thanks

Brink · Jul 20, 2024

Helping our customers through the CrowdStrike outage - The Official Microsoft Blog

On July 18, CrowdStrike, an independent cybersecurity company, released a software update that began impacting IT systems globally. Although this was not a Microsoft incident, given it impacts our ecosystem, we want to provide an update on the steps we’ve taken with CrowdStrike and others to...

blogs.microsoft.com

Hazel123 · Jul 20, 2024

antspants said:
A blow by blow running up-to-date blog of events today if anyone is interested. Including latest comments from CloudStrike CEO.

'Completely unprecedented' outage causes havoc with IT systems across globe — as it happened

A massive IT systems outage caused by issues with CrowdStrike software affects banks, airports, supermarkets and media companies across Australia and around the world.

www.abc.net.au

And a mention in there especially for John lol

Memories of 'Y2K'
By Greig Johnston
Quite a few people in our comments have mentioned the outage has shades of "Y2K".
For those of our readers too young to remember, that was when — in the year 1999 — there were widespread fears of a technological catastrophe because old computers formatted the date with just the final two digits of the year, rather than the full four-digit year.
There were worries this glitch would turn into a full-scale meltdown as the clock ticked over into the year 2000.
Companies rushed to make sure their systems were "Y2K compliant".
Thankfully the problems reported on January 1, 2000 were pretty minor.
But this was in a time when cash was still in widespread use, before "tapping" was the preferred currency.

I remember that - I got an extremely cheap flight out of it on 1 January 2000

Nobody wanted to fly that day.

A Guy · Jul 21, 2024

A Guy

Almighty1 · Jul 21, 2024

I am glad this happened earlier because I am flying on American Airlines on July 22, 23 and 31 and it would really be bad if this happened on those dates!

Almighty1 · Jul 21, 2024

neves said:
If that's the case - that's just "Marketing (deceit/manipulation)" at this point.

Cause on the technical side of things - this mistake proved to be the result of a kindergarten level of programing incompetence - revealing multiple serious issues with the state of the software and its programing. As in, we know now that the file/update responsible for this crash - was full of null characters:

View attachment 101822

It's what caused the crash, but that not the big concern - more like... WHY!? And... HOW (did it got to this point)?! What level of programing, testing and management did this "indie company" actually have - while handling a worldwide critical task?! Or is CrowdStrike handled by a student with a GitHub account - experimenting with some personal project (a bad student too - cause there's tons of programmers on GitHub - who wouldn't make this mistake simply by following basic protocols). Seriously...

1 - How was a null file pushed for updates - without being tested and signed (the most basic protocol - which every company with commercial products tends to have in place)? After all - this is a critical update - for a product used at an enterprise/public level (even by airports and hospitals) - and yet - somehow managed to miss both marks.

2 - Again, this being a critical update for a critical product - a second verification should take place on the client side as well. And yet, this verification - is either missing from the software or someone kid introduced a coding statement like: - if Null, Skip

3 - Why did the Kernel Module Crash while reading this file?!?!?!?!?!?!?! That shouldn't be possible in the first place (treating a null file as if it contains readable data to work with). That's quite literally - like Siri telling you to Turn Right off a cliff... and Crash - while you can see clearly on the GPS that there's no road there - but the software is so badly programed - that tries to forcefully read that data (and Crash as a result).

But hey, maybe their marketing team can sell sand in the desert. After all, at this point in time - it's less about the quality of a product - and more about the marketing/deception skill behind it.

Stock market pricing and reality are two different things.... GameStop and AMC wouldn't exist all this time if that was the case. What you said is only one analysis....

"The CrowdStrike bug was caused by a NULL pointer dereference in C++, a common error in memory-unsafe languages. This type of bug can lead to crashes or hashtag#security vulnerabilities, especially in privileged code like system drivers.

Key points:
1. C/C++ allow direct memory manipulation, making pointer errors common
2. NULL pointer dereferences cause crashes in privileged code (e.g. drivers)
3. Modern tools like AddressSanitizer can help catch these bugs
4. Proper error checking and testing are critical for system-level code

Rust is seen as a safer alternative because:
1. It prevents NULL and dangling pointer issues by design
2. Memory safety is enforced at compile-time
3. It offers performance comparable to C/C++

Rust is the future, period. C and C++ are becoming outdated. Rust excels with bulletproof memory safety and cutting-edge features.

For rock-solid, secure systems code, hashtag#Rust is the best option. The smart money's already on Rust—it's time for everyone else to wake up and smell the oxidation.

hashtag#cybersecurity hashtag#infosec hashtag#software hashtag#defense

Link to the screenshot: LinkedIn "

TairikuOkami · Jul 21, 2024

andrew129260 said:
So you control when your crowdstrike antivirus updates?

Of course, this literally happens at least once a year from various AV vendors.
One of the reasons why running AV on a critical machine is a terrible idea.
Updates on the network should be never applied simultaneously.

Botched CrowdStrike security update breaks Windows worldwide, causing BSOD and crashes

Well-known member

My Computers

Well-known member

My Computers

Well-known member

My Computers

Jack of all trades – master of none

My Computers

Microsoft 365 Basic

My Computer

Well-known member

My Computer

Banned

My Computer

Well-known member

My Computers

Well-known member

My Computers

Well-known member

My Computers

Well-known member

My Computer

Windows developer and admіn

My Computers

Well-known member

My Computer

Well-known member

My Computers

Administrator

My Computers

Well-known member

Memories of 'Y2K'​

My Computer

Righteous Dude

My Computers

Well-known member

My Computer

Well-known member

My Computer

Microsoft 365 Basic

My Computer

Similar threads

Memories of 'Y2K'