Botched CrowdStrike security update breaks Windows worldwide, causing BSOD and crashes

@andrew129260

You are evidently well informed. Have you any sense of how common it is for big companies to be totally in the cloud for their operations?
Do Fortune 500 companies use AWS or similar?

kelper said:

Would computers affected by the CrowdStrike bug be remotely bootable?

Click to expand...

Yes, of course. It's possible e.g. through SCCM with the combination of a PXE server, a DHCP server and a TFTP server, at least if the boot order on the client PCs was set to first attempt to PXE boot, i.e., to attempt to PXE boot before attempting to boot from the internal SSD. That is, at least if we can assume that these servers [that are required to make PXE boot possible] are still up and running of course. But then, if these servers no longer are up and running, a recovery plan should help solve that immediately (or hopefully it just might).

That plus the fact that the general concept of redundancy is always the main key, in a broad sense. If one server fails, other servers should be able to cope to the point where this negative impact can still be mitigated to some reasonable extent. Similarly, multiple different connections should be used to anticipate connection failures. Duplexing alone is very often not enough.

Duplexing is the concept of having, e.g., multiple NICs or multiple HBAs installed within the same PC or within the same server. So, while it is true that a RAID array gives you multiple redundant physical disks, it won't save your day if the RAID controller fails, which helps to explain why having multiple RAID controllers can be a good idea, but if the network connection fails or if the whole server goes down... well, you got the picture.

In short, I don't see how it would be possible that the only way to break out of the CrowdStrike bootloop on thousands of PCs would be to just press and hold the Delete key or F2 to enter the UEFI settings and change the boot order. Obviously instead, you want to change the boot order to how it's supposed to be, always before Crowd is Struck...

andrew129260 said:

So I was curious and asked my higher ups why we don't use PXE in my town hall meeting and they told me we don't use PXE because it's incredibly insecure. It doesn't meet their "zero trust" intiantive. They apparently use intune on their side while I just get machines shipped with images on them that I deploy per use case via PDQ.

I asked them what makes it insecure and didn't get far, so now to do some internet research.....

Click to expand...

One word? Incompetence.

andrew129260 said:

You still have to tell the machine to boot from pxe though?

Click to expand...

Besides changing the boot order again, there exist multiple other ways to prevent specific client machines from PXE booting. With SCCM, it is possible to organize specific client machines with dynamic collections and static collections. Dynamic collections are (WQL) query based. Templates can be used to create various queries for this purpose, but Powershell can also be used to leverage more flexibility in that regard.

Another example of how PXE boot can be prevented (again, on specific clients) is to configure the DHCP server in such a way that, during PXE, it assigns no IP address to the client, i.e., based on the MAC address of the client. So, for this to work, the MAC address of the client must be known. In short, if the client cannot reach both the PXE server and the TFTP server, then the attempt to PXE boot will be ended, and the client will attempt to boot the next item on the list that has been defined in the boot order of the client's BIOS/UEFI settings.

@hdmi

That is too complicated for little old me to understand. Can companies develop a strategy that would evade what happened on Friday? Even those that use CrowdStrike? If a machine Blue Screens, can it still be booted remotely?

kelper said:

@andrew129260

You are evidently well informed. Have you any sense of how common it is for big companies to be totally in the cloud for their operations?
Do Fortune 500 companies use AWS or similar?

Click to expand...

From what I have seen over the years and from what my friends tell me that are in the industry yes there has been a huge move to cloud for a ton of things. Especially for security.

kelper said:

@hdmi

That is too complicated for little old me to understand. Can companies develop a strategy that would evade what happened on Friday? Even those that use CrowdStrike? If a machine Blue Screens, can it still be booted remotely?

Click to expand...

From my understanding security providers would need to have more options for all updates patch management not just the program updates. But again, with security compliance, insurance compliance I am not sure how that could be handled in a way that would prevent things like this. Upgrade rings could help as suggested, but companies are terrified of security issues and compliance so a balance would have to be found. You cannot realistically test every single av update for issues.

No if a PC blue screens in a loop in the way the crowdstike one has, there isn't much you can do besides touching the machine in some way. You could have thin clients and a central server in some environments but that's not practical for most businesses. There is also the Intel management remote tool that is possible to do some things remotely but most companies don't use it due to security concerns. I'm sure with sccm and other stuff you can do more but that's outside my scope. According to HDMI you can do this via the pxe stuff, but you would need a server thats unaffected, which in our case everything was so that would not have helped.

hdmi said:

One word? Incompetence.

Besides changing the boot order again, there exist multiple other ways to prevent specific client machines from PXE booting. With SCCM, it is possible to organize specific client machines with dynamic collections and static collections. Dynamic collections are (WQL) query based. Templates can be used to create various queries for this purpose, but Powershell can also be used to leverage more flexibility in that regard.

Another example of how PXE boot can be prevented (again, on specific clients) is to configure the DHCP server in such a way that, during PXE, it assigns no IP address to the client, i.e., based on the MAC address of the client. So, for this to work, the MAC address of the client must be known. In short, if the client cannot reach both the PXE server and the TFTP server, then the attempt to PXE boot will be ended, and the client will attempt to boot the next item on the list that has been defined in the boot order of the client's BIOS/UEFI settings.

Click to expand...

Yeah I mean I see what you are saying but I also don't how that helps in the field. Most of our people are not in the building. Most of our laptops don't have Ethernet.

pseymour said:

PXE is quite old and has no encryption or auth mechanism. If doing Zero Trust, you would use UEFI’s HTTP boot, ideally HTTPS.

Click to expand...

They still call it PXE, but what they factually mean is iPXE. What is Preboot Execution Environment (PXE)?

kelper said:

@hdmi

That is too complicated for little old me to understand. Can companies develop a strategy that would evade what happened on Friday? Even those that use CrowdStrike?

Click to expand...

They most certainly can, and, they most certainly should have.

kelper said:

If a machine Blue Screens, can it still be booted remotely?

Click to expand...

It isn't the machine that Blue Screens. The thing that Blue Screens is called Windows. The thing that causes Windows to Blue Screen is the bad update from CrowdStrike. The thing that causes the bootloop is this. But keeping that checkbox enabled is also what's necessary for the machine to attempt to PXE boot after the Blue Screen occurs. Else, the machine will just stay forever frozen at the Blue Screen until either someone powers it down or someone manually restarts it. So, by keeping the AutoReboot option always enabled while at the same time also, in the boot order, keeping PXE boot always above the internal SSD that has Windows installed on it, it will be possible to allow the machine to, 1/ after the Blue Screen occurs, AutoReboot and, next, via PXE boot, 2/ receive a custom WinPE image that contains a startup script like how I earlier said (i.e., using Wpeinit and Startnet.cmd).

Once started, the script can delete the bad file, send a message over the network to inform the server that the operation was successful, wait for the server to acknowledge the message, and reboot. Upon receiving the message, the server can then prevent the client from PXE booting again, either by adjusting the DHCP server's settings (i.e., based on the client's MAC address) or by using Powershell to update the WQL query that the dynamic collection is based on (i.e., if using SCCM with WDS).

kelper said:

That is too complicated for little old me to understand. Can companies develop a strategy that would evade what happened on Friday? Even those that use CrowdStrike? If a machine Blue Screens, can it still be booted remotely?

Click to expand...

A blue screen is caused by a crash in kernel mode, and the crash is done intentionally by design. When an unexpected condition occurs in the kernel, the system is halted. In windows that a BSOD, in other OS's the same process happens (the intentional crash), but the colors and experience are difference.

In the case of crowdstrike, the machine would boot windows, but it loads the kernel mode driver for crowdstrike very soon in the boot process and that bug from crowdstrike caused a crash in kernel mode which resulted in the BSOD.

So the solution was to boot up the box to a minimal state, and then remove the corrupted channel file placed by Crowdstrike so that the crash in kernel mode didn't happen.

The problem on Friday is hard to prevent though as described by @hdmi because the reasons for the crash need to be known and understood to then take remediation steps in a script. Once understood, there are a variety of methods and tooling to get around the issue. So in most cases, this type of problem is going to happen again and it's not due to lacksadaisical system adminstation.

hdmi said:

So, by keeping the AutoReboot option always enabled while at the same time also, in the boot order, keeping PXE boot always above the internal SSD that has Windows installed on it, it will be possible to allow the machine to, 1/ after the Blue Screen occurs, AutoReboot and, next, via PXE boot, 2/ receive a custom WinPE image that contains a startup script like how I earlier said (i.e., using Wpeinit and Startnet.cmd).

Click to expand...

thats awesome for desktops and production. Would be cool if my company did that for them. That doesn't help for laptops though, which I would wager would be the most common type of machine across a company. They would need to be in the building on our network. And would also need to have ethernet ports. Meaning you would need to touch each machine anyway. And in the case of cloudstrike the server hosting the pxe would need to be unaffected as well. Not sure how this would affect bitlocker either.

pparks1 said:

The problem on Friday is hard to prevent though as described by @hdmi because the reasons for the crash need to be known and understood to then take remediation steps in a script. Once understood, there are a variety of methods and tooling to get around the issue. So in most cases, this type of problem is going to happen again and it's not due to lacksadaisical system adminstation.

Click to expand...

Agreed. There are certain things you can do better, but even the best companies went down with this.

In other news:

Further proof this could not be blocked by having update / patch management:

Configuration File Primer​

The configuration files mentioned above are referred to as “Channel Files” and are part of the behavioral protection mechanisms used by the Falcon sensor. Updates to Channel Files are a normal part of the sensor’s operation and occur several times a day in response to novel tactics, techniques, and procedures discovered by CrowdStrike. This is not a new process; the architecture has been in place since Falcon’s inception.


So yes, this could not have been prevented imho. And even if you had a team to test every update, there is not enough resources to test these on enough machines for long enough that come out multiple times a day.

Only the provider in my opinion would have the resources/time to do this.

Now you could have rings to deploy these updates out to workstations first and then production or critical systems last, but then the insurance compliance and security policy might prevent you from doing so. And what's the point if you dont have the latest updates for your AV? Millions of threats come out every day.

I don't know what the answer is, but I don't think testing every single AV update to release is it. Not with the countless ransomware and other attacks.

Your AV company has to have better QA. Microsoft needs to have better protection of a system when a blue screen occurs.

hdmi said:

They most certainly can, and, they most certainly should have.

It isn't the machine that Blue Screens. The thing that Blue Screens is called Windows. The thing that causes Windows to Blue Screen is the bad update from CrowdStrike. The thing that causes the bootloop is this. But keeping that checkbox enabled is also what's necessary for the machine to attempt to PXE boot after the Blue Screen occurs. Else, the machine will just stay forever frozen at the Blue Screen until either someone powers it down or someone manually restarts it. So, by keeping the AutoReboot option always enabled while at the same time also, in the boot order, keeping PXE boot always above the internal SSD that has Windows installed on it, it will be possible to allow the machine to, 1/ after the Blue Screen occurs, AutoReboot and, next, via PXE boot, 2/ receive a custom WinPE image that contains a startup script like how I earlier said (i.e., using Wpeinit and Startnet.cmd).

Once started, the script can delete the bad file, send a message over the network to inform the server that the operation was successful, wait for the server to acknowledge the message, and reboot. Upon receiving the message, the server can then prevent the client from PXE booting again, either by adjusting the DHCP server's settings (i.e., based on the client's MAC address) or by using Powershell to update the WQL query that the dynamic collection is based on (i.e., if using SCCM with WDS).

Click to expand...

That assumes someone who knows how the "f" to this. Obviously most didn't and don't. Even reading your example I'm scratching my head and wondering if I'd be able to do it.

Blown out of all proportion - I've been running 7 different systems, using all sorts of services and didn't notice ANY issues.

Still, let's not deprive the media of their weekly fix of sensationalism, eh?

andrew129260 said:

thats awesome for desktops and production. Would be cool if my company did that for them. That doesn't help for laptops though, which I would wager would be the most common type of machine across a company. They would need to be in the building on our network. And would also need to have ethernet ports. Meaning you would need to touch each machine anyway.

Click to expand...

PXE over WiFi is a vendor-specific feature that isn't very common, but some laptops have it. E.g., the HP EliteBook 840 G7 and the HP ProBook 440 G8. If you have admin rights, an alternative approach would be to use OSDCloud to be able to download a custom WinPE over the internet via WiFI.

andrew129260 said:

And in the case of cloudstrike the server hosting the pxe would need to be unaffected as well.

Click to expand...

If all the servers are forever stuck in a bootloop, then like I earlier tried to point out, every IT manager (every sane IT manager, that is) should already have a recovery plan ready before it happens, complete with compelling evidence built on valid test results to indicate that the recovery plan is reliable and robust.

andrew129260 said:

Not sure how this would affect bitlocker either.

Click to expand...

It should work.

andrew129260 said:

Agreed. There are certain things you can do better, but even the best companies went down with this.

Click to expand...

If they went down with this, it only proves that they weren't truly the best. Everyone can make mistakes, but letting yourself get locked out of two boatloads of PCs at once? Yeah...

andrew129260 said:

In other news:

CrowdStrike says significant number of devices back online after global outage

But experts says full recovery from Friday’s IT failure could take weeks

www.theguardian.com

Click to expand...

The article says they're working on a solution to make it reboot faster. I was thinking maybe it will be a mini-sized bootable image.

mingle said:

Blown out of all proportion - I've been running 7 different systems, using all sorts of services and didn't notice ANY issues.

Still, let's not deprive the media of their weekly fix of sensationalism, eh?

Click to expand...

And do you pay for cloudstrike as a third party software for endpoint protection? If not, then you would not have seen any fallout from the issue.

We had nearly 1000 servers down and thousands of employee workstations to fix from this issue. It wasn't sensationalism at all

steveg said:

That assumes someone who knows how the "f" to this. Obviously most didn't and don't. Even reading your example I'm scratching my head and wondering if I'd be able to do it.

Click to expand...

The official docs about Intune contain a few interesting articles that explain in fair detail how booting from a PXE server works.

A Guy

hdmi said:

If all the servers are forever stuck in a bootloop, then like I earlier tried to point out, every IT manager (every sane IT manager, that is) should already have a recovery plan ready before it happens, complete with compelling evidence built on valid test results to indicate that the recovery plan is reliable and robust.

Click to expand...

They were VM's so yes we implemented our backup snapshots and everything was fine. But that still takes time.

We also have redundancy for each thing with a separate machine as well, but they both had crowdstrike.

Good to know about the pxe over wifi. Never heard of that being a thing.

mingle said:

Blown out of all proportion - I've been running 7 different systems, using all sorts of services and didn't notice ANY issues.

Still, let's not deprive the media of their weekly fix of sensationalism, eh?

Click to expand...

Your running CrowdStrike on 7 computers and didn't have any problems. All I can say is Amazing!

Winuser said:

Your running CloudStrike on 7 computers and didn't have any problems. All I can say is Amazing!

Click to expand...

True, didn't say anything about those systems using CrowdStrike, unless we're supposed to assume that. And we know what that word really means.