Botched CrowdStrike security update breaks Windows worldwide, causing BSOD and crashes

hdmi · Jul 19, 2024

andrew129260 said:
Organizations have things set up that are out of your hands man. We don't have pxe anything. I can only change what I can change.

Also our systems are locked down and don't have that option to boot from that. It only allows boot from a locked bitlocker drive. I cannot alter the bios.

And even if we had pxe it would still be wiping the system or restoring from a backup, which you still need to touch each machine to tell it to boot from pxe.

I only have the tools I have.

Also clients with laptops that are not in the office, good luck with that.

That's exactly my point. You are toolless because they are clueless.

Configure a PXE server to load Windows PE (Windows 10) - Windows Deployment

This article describes how to configure a PXE server to load Windows PE so that it can be used with an image file to install Windows 10 from the network.

learn.microsoft.com

Wpeinit and Startnet.cmd: Using WinPE Startup Scripts

learn.microsoft.com

RJARRRPCGP · Jul 19, 2024

antspants said:
I didn’t notice a thing and I’ve been computing online all day.

Yep, no problems and I usually let my PC idle all night long. So I would have known if Windows was in fatal-kernel-error-mode.

andrew129260 · Jul 19, 2024

RJARRRPCGP said:
Yep, no problems and I usually let my PC idle all night long. So I would have known if Windows was in fatal-kernel-error-mode.

thats because you dont use crowdstrike :)

RJARRRPCGP · Jul 19, 2024

andrew129260 said:
thats because you dont use crowdstrike :)

I then quickly knew it was a CrowdStrike issue, even when I went to a video reporting this, because I wasn't having new crash issues with the July patch-Tuesday.

antspants · Jul 19, 2024

andrew129260 said:
I understand your opinion I really do. But it's not a perfect world, and I cannot change corporate policy. Many companies are not going to pay to have two different antivirus companies so that this doesn't happen. Or have the team or resources to test each antivirus update. Or if the antivirus even offers a management console to begin with to deny or approve updates.

I know it seems easy to just have backups in place and just have a solution, but that is not always possible. Also not everyone in IT is as smart as a lot of the people in here. I know I am not. I do what I can with what I have. And that is all I can do. I learn as much as I can.

Corporate especially is a whole other animal. Trying to get them to do the right thing or have good procedures is like pulling teeth.

Maybe there is things they could do, but I just don't think its practical enough. stuff happens. And unfortunately many people and organizations it happened today.

I’m not unsympathetic to your troubles, Andrew

It’s a horrible status.

hdmi · Jul 19, 2024

andrew129260 said:
thats because you dont use crowdstrike :)

lmao

andrew129260 · Jul 19, 2024

I wish I could laugh emoji that post way more than possible. oh yeah, and cry

RJARRRPCGP · Jul 19, 2024

andrew129260 said:
I wish I could laugh emoji that post way more than possible. oh yeah, and cry

At least when I made a big whoops back in 1995 or 1994, most likely, (maybe 1996? Even when I have a doubt about that!) when I ended up deleting a file that turned out to be for Novell Netware on a Windows 3.1 computer, IIRC, back when I still hardly knew about computers, nothing got taken out anything campus-wide. Just that the PC for the group home I was in, couldn't log in! I suddenly saw an error message about a Netware file being missing, when in Windows 3.1, IIRC.
I learned and made sure to never make a whoops like that again!

glockster · Jul 19, 2024

Crowdstrike must use the same QA team as Microsoft! None.

hdmi · Jul 19, 2024

Las Vegas

A Guy · Jul 19, 2024

RD Mercer said:
Can't wait to see what CrowdStrike does with all of the electronic voting machines on election day here. I wonder is this a test run for the "UPDATE” they’re going to install on the voting machines this November?

My Crowdstrike stock took a hit today, down 11%...still up 128% since I bought it

A Guy

ian50 · Jul 20, 2024

@andrew129260

This was the fix:

del /f /q "C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys"

After doing that everything was back to normal.

Yes, easy to understand, EXCEPT:

if, as you surely point out, these CrowdStrike updates are silently automated, then having deleted the little beggar and rebooting, you must be in exactly the same situation because naughty sys file will auto load again. That is, a corrupted update is again causing continuous Blue Screening. Unless, either CrowdStrike has replaced the corrupt sys file so the replacement is now auto loaded and hunky dory eats ice cream; or after deleting the delinquent but before reboot you disconnect from the internet, following which you have an uncorrupted PC working flawlessly but unable to perform normal net contact.

andrew129260 · Jul 20, 2024

RJARRRPCGP said:
At least when I made a big whoops back in 1995 or 1994, most likely, (maybe 1996? Even when I have a doubt about that!) when I ended up deleting a file that turned out to be for Novell Netware on a Windows 3.1 computer, IIRC, back when I still hardly knew about computers, nothing got taken out anything campus-wide. Just that the PC for the group home I was in, couldn't log in! I suddenly saw an error message about a Netware file being missing, when in Windows 3.1, IIRC.
I learned and made sure to never make a whoops like that again!

Oh jeez, I remember zenworks from novell. Used that at a school district at my last job before they switched over to AD. It was terrible.

ian50 said:
@andrew129260

This was the fix:

del /f /q "C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys"

After doing that everything was back to normal.

Yes, easy to understand, EXCEPT:

if, as you surely point out, these CrowdStrike updates are silently automated, then having deleted the little beggar and rebooting, you must be in exactly the same situation because naughty sys file will auto load again. That is, a corrupted update is again causing continuous Blue Screening. Unless, either CrowdStrike has replaced the corrupt sys file so the replacement is now auto loaded and hunky dory eats ice cream; or after deleting the delinquent but before reboot you disconnect from the internet, following which you have an uncorrupted PC working flawlessly but unable to perform normal net contact.

Yeah the update was patched on their end that updated the definitions & driver. You just had to do the fix and then once they booted up and got the newest update all was well. Took only a few minutes of being logged in for the new update to apply and fix it.

RJARRRPCGP · Jul 20, 2024

andrew129260 said:
Oh jeez, I remember zenworks from novell. Used that at a school district at my last job before they switched over to AD. It was terrible.

Yeah the update was patched on their end that updated the definitions & driver. You just had to do the fix and then once they booted up and got the newest update all was well. Took only a few minutes of being logged in for the new update to apply and fix it.

I went from being a big-time computer dummy to actually knowing how to fix an OS in 1997. I ran DOS like a boss in 1997, 1998 and 1999!
Same with Windows! Even when I thought Windows 95 was a bit tricky at first.

I suddenly was trying hard to become a system software fixer, and it paid off!

My "late-bloomer-syndrome" made me behind in 1994, 1995 and 1996.

A Guy · Jul 20, 2024

How Windows 3.1 is saving Southwest's butt | Digital Trends

Southwest hasn't been impacted by the CrowdStrike outage, and that's reportedly because it's still running Windows 3.1.

www.digitaltrends.com

A Guy

jimbo45 · Jul 20, 2024

Well if this is what happens when it's "just a simple software error" then imagine what a real concerted attack could do.

All this outsourcing etc exacerbates this problem to the nth degree. Maybe businesses should not always be striving to get the cheapest possible cost solutions but be slightly more responsible towards their customers etc. What happens if you are trying to check out of a hotel room in a foreign country where the banking / credit card system has failed or you can't get even small amounts of cash from machines. (Amex is OK though --their system was still working -- probably worth the yearly fee for a decent credit card as I've never known it in years to go down).

As for Windows itself -- shouldn't corporations / work places keep backup images so if Windows doesn't boot properly the last working copy gets automatically restored. Perhaps time also for big corporations to switch to Unix / Linux servers and for their Windows clients serve them as Windows VM's where the main internet connections is via the UNIX / LINUX gateway / proxy so the security could be based on the Unix type system which is far more robust than Windows.

Cheers
jimbo

andrew129260 · Jul 20, 2024

For more information on how this happened see this Twitter thread

https://twitter.com/x/status/1814376668095754753

andrew129260 · Jul 20, 2024

jimbo45 said:
As for Windows itself -- shouldn't corporations / work places keep backup images so if Windows doesn't boot properly the last working copy gets automatically restored

Windows was booting up correctly. The bsod was happening a little bit after the logon screen appeared.. So automated systems that looks for this including windows itself could not fix the issue because technically it booted windows just fine.

This was a low level driver because it was an antivirus. So it unfortunately has more capabilities of bringing the system down vs just a regular app or driver.

These low level types of drivers need full admin rights to install and must be digitally signed as well.

jimbo45 · Jul 20, 2024

andrew129260 said:
Windows was booting up correctly. The bsod was happening a little bit after the logon screen appeared.. So automated systems that looks for this including windows itself could not fix the issue because technically it booted windows just fine.

This was a low level driver because it was an antivirus. So it unfortunately has more capabilities of bringing the system down vs just a regular app or driver.

These low level types of drivers need full admin rights to install and must be digitally signed as well.

It still seems a bit of a risky method judging by the numbers of machines effected. Surely there has to be a better way of managing these updates --- perhaps - certainly on Unix type systems one could "replay the system logs" which would contain details of client machine logons (windows or otherwise) and start roll back jobs for the machines that crashed. Easy enough to record IP of each client, ping it and if it fails then initiate recovery. You can use all sorts of Linux / Unix utilities to recover Windows images - too e.g dd, rsync etc. You don't have to be an I.T genius to work something simple out such as this.

I still though don't understand a) why this stuff was rolled out on a Friday morning, b) there weren't immediate backup servers available, and c) robust testing obviously wasn't done

Once the windows update had been loaded on to the client machines then obviously you are in BIG trouble since a lot of these machines could only be fixed manually --- and many companies have outsourced entire I.T depts so getting individuals available to fix the machines will take far longer than if the technicians were in house. Definitely new procedures need to be designed and implemented.

Cheers
jimbo

andrew129260 · Jul 20, 2024

This is a good read on the events and what happened

2024 CrowdStrike-related IT outages - Wikipedia

en.wikipedia.org

Now I can finally sleep.

Botched CrowdStrike security update breaks Windows worldwide, causing BSOD and crashes

Jack of all trades – master of none

My Computers

Well-known member

My Computers

Well-known member

My Computers

Well-known member

My Computers

Like a rolling drone.

My Computers

Jack of all trades – master of none

My Computers

Well-known member

My Computers

Well-known member

My Computers

Well-known member

My Computers

Jack of all trades – master of none

My Computers

Righteous Dude

My Computers

Well-known member

My Computer

Well-known member

My Computers

Well-known member

My Computers

Righteous Dude

My Computers

Well-known member

My Computer

Well-known member

My Computers

Well-known member

My Computers

Well-known member

My Computer

Well-known member

My Computers

Similar threads