Planes Grounded, Businesses Disrupted Due to Tech Issue

mental health is nothing to be joked about, even with our AI friends.. in case any of them are listening...

we need a I identify as a quantum or super computer meme...

 

I better be able to play CFB '25 when I get home

 

It is. Those running on a pure Linux system were unaffected with the same CrowdStrike software.

 

Located the source of the issue!

 

Test run for election night.

 

Why IT folks never want to do updates on critical systems. But windows is convenient…..lol..

So will CrowdStrike be liable for losses?

 

From the details I've seen, in their latest push, one of the files, a windows .sys, was "empty". Not empty as in no contents, but rather the hex contents were all zero - 0x00000000 (NULL). That lead to a null pointer de-reference, crashing things nicely during boot.

Based on my decades of experience in software, this is almost guaranteed to be a human or process error in their release validation. Not a hack, as someone suggested upthread.

This will do significant damage to Crowdstrike's reputation...

 

I own a fair amount of crowdstrike stock. Not a great day.

 

Been a tough week across the board. Deployed some dry powder today. 35% of cash back in. This drop isn't warranted

 

time to buy more

 

Most tragically I couldn't order pizza from my local spot because their system was down.

 

I'm not a dev but I was wondering how this was deployed. Given how big their customer base is, I'd imagine it was tested thoroughly and everything was fine until someone, or some system, accidentally pushed out something other than was to be the production version?

I would think there was a process to recieve the update the same way the customers would receive it and test that in-house as well before putting it out into the wild. I guess the way I'm imaging it is like SolarWinds finishing their final build, but not realizing what was hosted for their customers was different from their build, something as simple as a hash comparison would have found. A disconnect of what's available internally and what's provided externally. The difference is I don't think this was malicious.

 

I’m sure we will get a full RCA in the next few days/weeks. But hard to believe this not only made it to production, but was widely deployed.

If what’s on twitter is correct, this was a basic programming error, and some mouth breathers are trying to blame DEI of course. But this was a failure on so many levels. No one developer should be able to cause this much damage. This is why you have code reviews, automated testing, and canary deployments. This was a leadership failure, and heads should roll.

 

I'm not familiar with their software, but I would think that large organizations with a ton of client devices would have a similar setup to a WSUS where their devices would hit internal servers to pick up patches instead of going out to Microsoft for them. Admins routinely sit on updates as they come out and test them. Only after testing are the approved updates listed on the WSUS for download by the client machines. You don't have to be in IT to know that Windows Updates can, have, a will forever cause issues. Being able to pick and choose patches before they're deployed to the vast majority of computers on the network is invaluable.

If there is a setup like this, and these companies had a chance to sit on this Crowdstrike update long enough to test it out before deployment, but didn't, then they deserve some blame, too.

 

I’m sorry for your loss