Recent searches
Search options
Upguard have published a list of companies they say are impacted by the CrowdStrike 'Global IT Outage', based on public reporting.
https://www.upguard.com/crowdstrike-outage
Edit: obviously it’s missing most companies as most companies aren’t disclosing publicly.
If anybody wonders what the file that took down 8.5 million Windows systems looks like.. it was 41kb in size. The only validity checking I can see CrowdStrike driver does is to check the first few bytes match the pattern seen in the screenshot before loading and executing.
The US Department of Transport has opened an investigation into Delta over the disruption related to CrowdStrike incident.
Good luck to the CrowdStrike account manager for Delta.
The initial Post Incident Review is out from CrowdStrike. It’s good and really honest.
There’s some wordsmithing (eg channel updates aren’t code - their parameters control code).
The key take away - channel updates are currently deployed globally, instantly. They plan to change this at a later date to operate in waves. This is smart (and what Microsoft do for similar EPP updates).
https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/
By ‘this is smart’ I mean ‘this is smart… now’. Obviously they shouldn’t have been globally, simultaneously deploying kernel driver parameter changes across all customers: it was waiting to go wrong.
They still are btw, as it will take a while to engineer the correct way of doing it.
On insurance and CrowdStrike, Parametrix claim amongst just the Fortune 500 companies, they are facing $5.4bn in losses, of which around 10% will be covered by insurance.
https://www.theguardian.com/technology/article/2024/jul/24/crowdstrike-outage-companies-cost
CrowdStrike have won this year's Pwnie Award for Epic Fail, which will please @qwertyoruiop.
If you want to know something crazy:
- This year TCS migrated their EDR to CrowdStrike
- Then they announced a strategic partnership with CrowdStrike
- Then they lost all their systems
- They’re just finishing recovery today, 6 days in
- Then they got a $10 Uber Eats voucher
- …which got cancelled due to Uber flagging CrowdStrike’s account as fraudulent
Questions for your EDR providers (do not assume they are experts in availability):
- What are your different update processes?
- How do you test them?
- Do you dogfood test them?
- Do you roll them out in waves? What are the details, eg what percentages and when?
- Do you monitor failures and roll back?
CrowdStrike staff members are selling CrowdStrike monopoly sets they were given on eBay.
CrowdStrike filed at 8-K with the SEC on July 22nd for a cybersecurity incident. https://www.board-cybersecurity.com/incidents/tracker/20240722-crowdstrike-holdings-inc-cybersecurity-incident/
Almost a week in, CrowdStrike say 97% of devices are back online. https://www.axios.com/2024/07/25/crowdstrike-97-percent-systems-online
Microsoft are talking about changes to Windows after the CrowdStrike incident. Good.
https://www.theverge.com/2024/7/26/24206719/microsoft-windows-changes-crowdstrike-kernel-driver
There’s a really good discussion on @riskybusiness’s YouTube show about the CrowdStrike incident.
About the 3 minute mark @alex made me realise I was far too kind to CrowdStrike. He rightly rips them apart.
Delta are looking to sue CrowdStrike and Microsoft. HT @hrbrmstr
Re the Delta case - the lawyer they’ve hired successfully sued Microsoft previously on behalf of the US government, and the decision was upheld on appeal too. The ruling almost lead to the breaking up of Microsoft.
The following US government backed out of the case.
Bill Gates said at the time the lawyer was “out to destroy Microsoft”.
So there’s a chance here the CrowdStrike incident may end up having implications across vendor industry around warranties etc, we’ll see.
Replacing an XDR platform at scale takes some time, so if you’re wondering what the translation of Elon’s tweet about Crowdstrike is:
Elon: can we replace Crowdstrike?
Somebody: yes, we’ll begin looking into it but..
Elon: job done
Of course.. given how the Twitter takeover happened maybe he just got them to uninstall it and #yolosec
Delta’s CEO has confirmed they plan to take legal action against CrowdStrike after incurring a $500m loss
6 minute video interview: https://www.cnbc.com/2024/07/31/delta-ceo-crowdstrike-microsoft-outage-cost-the-airline-500-million.html
CrowdStrike made a net loss of $845m between 2018 until this year, and has taken on $743m of debt during this period.
Spirit Airlines in the US anticipates a $7.2 million hit to its third-quarter operating income due to operational disruptions caused by the CrowdStrike incident, which forced the carrier to cancel 470 flights.
Here's the Delta boss on his thoughts about the CrowdStrike incident.
They had 40k Windows Server boxes alone, all with BitLocker full disk encryption enabled, all of which wouldn't boot and weren't fixable without manually unlocking BitLocker. That had gone all in with CrowdStrike + Microsoft's most premium offerings.
He has a really good point about how tech companies have become obsessed with growth as their only metric of success, and customer satisfaction is not on the radar.
There's a really mad moment in that interview where they ask them what assistance CrowdStrike have offered, and he essentially says nothing, not even a lunch voucher.
What a time to be alive.
CrowdStrike complained to Cloudflare about a CrowdStrike parody site… and Cloudflare took it down. Without a court order. https://clownstrike.lol/crowdmad/
Cloudflare recently announced they have become a strategic partner with CrowdStrike: https://www.cloudflare.com/en-gb/press-releases/2024/crowdstrike-and-cloudflare-announce-expanded-strategic-partnership-to-secure/
Additionally to loop this in, CrowdStrike submitted a takedown for a parody label (they’ve since rescinded it after being called out).
We’ve reached the part of the brand cycle where people are using CrowdStrike as an excuse https://www.theverge.com/2024/8/2/24212298/mrbeast-beast-games-crowdstrike
360 takes a look at the Crowdstrike kernel drivers - finds they implement an eBPF like system, contain a wide attack surface, don’t check validity of update files (eg no signing of updates) and claim they contain conditions for LPE and RCE vulnerabilities. https://mp.weixin.qq.com/s/uD7mhzyRSX1dTW-TMg4UhQ
Before people write this off as ‘the Chinese’, I’ll give you a hint: there really, really should be security research about the security of security products across all vendors. I’ve seen things.
Previously on Crowdstrike Falcon vulnerability research, check out this timeline where they tried to use NDAs to avoid disclosure, then fixed it without telling anybody. https://modzero.com/modlog/archives/2022/08/22/ridiculous_vulnerability_disclosure_process_with_crowdstrike_falcon_sensor/index.html
EFF are calling for antitrust action after the CrowdStrike incident https://www.eff.org/deeplinks/2024/07/crowdstrike-antitrust-and-digital-monoculture
Bloomberg report a vast majority of the CrowdStrike losses reported by customers will be judged by insurance as not covered by policies. https://www.bloomberg.com/news/articles/2024-08-02/billions-in-damages-from-crowdstrike-outage-to-go-uninsured
CrowdStrike are publicly threatening their customer, Delta. https://www.theverge.com/2024/8/5/24213521/crowdstrike-refutes-blame-delta-outage-litigation
I've written up a bit about CrowdStrike's latest bold strategy.
Microsoft have now queued up to try publicly throw their customer under the bus, claiming (without evidence) Delta’s CrowdStrike woes were due to non-Windows systems. The CrowdStrike issue only impacted Windows systems so I hope somebody at Microsoft knows what they are doing.
https://www.theverge.com/2024/8/6/24214371/microsoft-delta-letter-crowdstrike-response-comments
If anybody wants the subtext of what is happening here, CrowdStrike and Microsoft both really do not want to get sued by Delta and have it go to court as it would potentially be explosive for both orgs and the wider security industry.
The customers are always plebs to be milked, as is status quo.
CrowdStrike incident root cause analysis is out.
Overall, good… but.
It is very verbose but doesn’t say much. Some of the wording will confuse people - eg it talks about rings (waves) in a way which makes you think it is already implemented. It isn’t. They’re saying they plan to implement it later.
Channel updates weren’t tested on a real Windows PC prior to deployment, they relied on automated bespoke code testing. They don’t mention that and it’s the real reason.
Risky Business take on CrowdStrike root cause report is good.
You can see the confusion the report provides in this discussion I think, eg some of the things are talked about as being implemented - but they’re down as findings for improvement. It’s the way the report is worded, to make you believe certain things existed.. that don’t yet.
Really good piece about CrowdStrike (technically CSC) misusing DMCA takedown notices over trademark disputes.
CrowdStrike probably want to have a word with CSC about this and Cloudflare should tighten process as DMCA isn’t supposed to be used for this. I know CSC do it.. but they shouldn’t be.
Wider point: cyber industry abusing process in takedowns.
CrowdStrike have responded to two claimed vulnerabilities in CrowdStrike Falcon, including one made by a former staff member: https://www.crowdstrike.com/blog/tech-analysis-addressing-claims-about-falcon-sensor-vulnerability/
There may be more to come on this one..
CrowdStrike vs Delta vs Microsoft continues to play out in public, now with SEC filings
The Germany government interviewed over 300 companies to find out the CrowdStrike impact to their orgs. Over 60% were impacted, and around half had to halt all operation, for an average of 10 hours. https://www.bsi.bund.de/DE/Service-Navi/Presse/Pressemitteilungen/Presse2024/240919_BSI-bitkom_Crowdstrike-Umfrage.html
The BBC have an in depth look at the impact of the CrowdStrike ‘global IT outage’ aka happy little non-cyber availability incident
It’s a really good article. Held up cancer care etc. https://www.bbc.com/news/articles/cr54m92ermgo
Delta have filed their lawsuit against CrowdStrike, accusing the firm of gross negligence in their testing regime.
https://www.wsj.com/business/airlines/delta-sues-crowdstrike-over-july-operations-meltdown-099ad8fa
CrowdStrike are now counter suing their own customer 
I really hope this heads to discovery and trial as Delta are correct about the testing on one system thing. CrowdStrike know that. They just aren’t admitting it.
Microsoft are making further changes to try to prevent another CrowdStrike moment https://www.theverge.com/2024/11/19/24299873/microsoft-windows-resiliency-initiative-crowdstrike-incident
CrowdStrike says customers who left “won’t be missed” HT @metacurity
Congrats to CrowdStrike for being top right of the latest magic quadrant, for world's biggest cyber attacks!
@GossiTheDog 
the internet will internet. Unbelievable how many PR people still don't understand the streisand effect.k
@GossiTheDog I love how so many of us have said "I didn't know that parody site existed until I saw the news about the DMCA"
Have CSC never heard of the Streisand Effect?
@GossiTheDog I've read one of the reports they are talking about. I didn't post about it here, because I found its arguments tenuous at best, although I could have misunderstood them, given that I can't read Chinese and automatic translation usually makes a mess of it.
However, there are places in CrowdStrike's refutation of it that make me go "Hmmm".
@GossiTheDog 1) Certificate pinning will ensure integrity of the delivered content but would do nothing to preserve this integrity of the content once stored on the disk. So, yeah, a MitM attach might not be possible but it's a bit disingenuous to talk about it in the same breadth while "refuting" that the downloaded files can be tampered with on the disk.
@GossiTheDog 2) "A malicious network proxy might block traffic destined for CrowdStrike servers" - no, schmucks, if you respect the proxy settings, a malicious proxy CAN block traffic to and from CrowdStrike's servers.
@GossiTheDog 3) Checksum validation helps ensuring the integrity of the delivered content but is not robust enough to ensure the integrity of the content already delivered, because whoever modifies it on the disk, can modify the checksum too, despite the handwavy "it's stored in a secure location". Like, where? Surely not in the TPM?
@GossiTheDog 4) Yeah, bypassing the ACLs would require that the attacker gains admin rights first - but how many users run with admin rights and isn't the proper question to ask "what harm can the attacker do to and with our product if he gains such rights"?
@GossiTheDog 5) The question of whether the regular expressions that CrowdStrike's product uses are Turing-complete or not is tenuous at best. We'd need to know exactly what kind of pattern matching they use. Standard regexes are indeed not Turing-complete, see
https://www.perlmonks.org/?node_id=809842
But once you get to context-free grammars and higher in the Chomsky hierarchy, things start getting dicer... Anyway, I can't answer this question authoritatively without more information about their searching language.
@GossiTheDog 6) Even assuming a very simple pattern matching language, we have to ask ourselves "what damage can the attacker cause if he can control the pattern?". Is CrowdStrike's product detection-only - or does it try to do any kind of removal/quarantine/etc.?
Surely you wouldn't want the attacker to supply a search pattern that would result in the removal of the contents of the Windows System32 directory?
@GossiTheDog Anyway. My point isn't that their product is vulnerable - my point is that their refutation of some of the points raised by the researchers is tenuous in some places and I would have preferred a better and more clear wording.
@GossiTheDog
They have the benefit of the doubt until a wild PoC appears
@GossiTheDog
The did deliver. On a Friday. They didn't delay the update to test it or other such nonsense. They delivered.
@GossiTheDog I mean technically they were true to their word; they promised to be a "provider that delivers without compromise" and they sure didn't compromise on delivering their borked update to a global fleet of devices 
@GossiTheDog “A rogue software update” sounds like it became sentient. Or the work of corporate espionage.
@GossiTheDog It should have been the watershed moment that finally spurred thousands of IT managers to move away from the MS monoculture. But itwasn't.
@GossiTheDog that one’s been coming
@GossiTheDog the time to question a business’s practices is when you are vetting them. They had the price you wanted, you got what you were concerned with.
@GossiTheDog it was pretty gross
I wondered when that would happen.
@GossiTheDog I heard from someone in the know that Crowdstrike had been _begging_ Delta to upgrade certain parts of their infrastructure, that they knew something like this could happen any day and tried to warn Delta about it. That Delta was uniquely lax in keeping up its technology.
(doesn't assuage Crowdstrike from partial blame, just saying)
Itll be a very interesting court case if it goes that far. Like SCO vs. everyone level tedious (but technically verya interesting)
Dear corporations,
You can proceed with all of this insanity like a pack of rabid bonobos with some kind of weird fetish,
-- OR --
you could JUST NOT RUN WINDOWS.
@GossiTheDog ... this is not the way to secure systems. These business models totally rely on FUD and technological ignorance.
@GossiTheDog Probably hoping to settle quick with that counter suit and avoid the discovery.
@GossiTheDog Microsoft is busy creating future Microsoft moments.
@GossiTheDog @metacurity that’s not how I understood that article.
"CEO George Kurtz was more upbeat, telling investors that CrowdStrike's products are what customers want and need right now as they try to defend against increasing cyber crime.
"I'm encouraged by the conversation that I'm having with our largest customers and a reflection on the fact that they realize that we have the best tech in the industry and the ability to stop breaches," he said, adding that customers are mostly sticking around.
Some small managed services providers have bailed, but the CEO opined that they won't be missed."
@metacurity @GossiTheDog the way you picked that bit of text painted a more dismissive picture than the full quote:
"Some small managed services providers have bailed, but the CEO opined that they won't be missed."
@GossiTheDog @metacurity they really don't seem to like their customers. Too big to fail?
@GossiTheDog @metacurity This is giving strong "you can't fire me, I quit!" energy.
@GossiTheDog @metacurity I think this was misunderstood in the article.
@mbmy @GossiTheDog I can see how the writer interpreted the small MSP space "really doesn't matter" as "won't be missed."
But I will delete my toot that quoted the article directly to not create any confusion.
@metacurity @GossiTheDog Yeah I think what they were insinuating is that "for small MSPs, the product they select doesn't matter as much because they have less barrier to move between vendors"
@mbmy @GossiTheDog FWIW, the relevant section begins at around 44:10 in the webcast, and I hear Kurtz's tone of voice saying, "It really doesn't matter" that some MSPs churned away as they "won't be missed."
I would have interpreted it the same way as the Reg writer. But he didn't technically say "won't be missed."
@GossiTheDog Hoist that jersey up to the rafters
FYI, CrowdStrike's share price of ~ $415 currently exceeds its high of ~ $392 prior to the the July incident.
@GossiTheDog and they didn't even need to pay Gartner to be considered! It's a Win Win WIn!
@GossiTheDog I've met engineers from Crowd strike and they were really cool people. Meeting their manager on the other hand...