Back

Upguard have published a list of companies they say are impacted by the CrowdStrike 'Global IT Outage', based on public reporting.

https://www.upguard.com/crowdstrike-outage

Edit: obviously it’s missing most companies as most companies aren’t disclosing publicly.

If anybody wonders what the file that took down 8.5 million Windows systems looks like.. it was 41kb in size. The only validity checking I can see CrowdStrike driver does is to check the first few bytes match the pattern seen in the screenshot before loading and executing.

The US Department of Transport has opened an investigation into Delta over the disruption related to CrowdStrike incident.

Good luck to the CrowdStrike account manager for Delta.

The initial Post Incident Review is out from CrowdStrike. It’s good and really honest.

There’s some wordsmithing (eg channel updates aren’t code - their parameters control code).

The key take away - channel updates are currently deployed globally, instantly. They plan to change this at a later date to operate in waves. This is smart (and what Microsoft do for similar EPP updates).

https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/

By ‘this is smart’ I mean ‘this is smart… now’. Obviously they shouldn’t have been globally, simultaneously deploying kernel driver parameter changes across all customers: it was waiting to go wrong.

They still are btw, as it will take a while to engineer the correct way of doing it.

On insurance and CrowdStrike, Parametrix claim amongst just the Fortune 500 companies, they are facing $5.4bn in losses, of which around 10% will be covered by insurance.
https://www.theguardian.com/technology/article/2024/jul/24/crowdstrike-outage-companies-cost

CrowdStrike have won this year's Pwnie Award for Epic Fail, which will please @qwertyoruiop.

If you want to know something crazy:

- This year TCS migrated their EDR to CrowdStrike
- Then they announced a strategic partnership with CrowdStrike
- Then they lost all their systems
- They’re just finishing recovery today, 6 days in
- Then they got a $10 Uber Eats voucher
- …which got cancelled due to Uber flagging CrowdStrike’s account as fraudulent

CrowdStrike are… having a week.

Questions for your EDR providers (do not assume they are experts in availability):

- What are your different update processes?
- How do you test them?
- Do you dogfood test them?
- Do you roll them out in waves? What are the details, eg what percentages and when?
- Do you monitor failures and roll back?

CrowdStrike staff members are selling CrowdStrike monopoly sets they were given on eBay.

CrowdStrike filed at 8-K with the SEC on July 22nd for a cybersecurity incident. https://www.board-cybersecurity.com/incidents/tracker/20240722-crowdstrike-holdings-inc-cybersecurity-incident/

Almost a week in, CrowdStrike say 97% of devices are back online. https://www.axios.com/2024/07/25/crowdstrike-97-percent-systems-online

Microsoft are talking about changes to Windows after the CrowdStrike incident. Good.

https://www.theverge.com/2024/7/26/24206719/microsoft-windows-changes-crowdstrike-kernel-driver

There’s a really good discussion on @riskybusiness’s YouTube show about the CrowdStrike incident.

About the 3 minute mark @alex made me realise I was far too kind to CrowdStrike. He rightly rips them apart.

https://youtu.be/EGRqtscp4eE

Delta are looking to sue CrowdStrike and Microsoft. HT @hrbrmstr

https://www.cnbc.com/2024/07/29/delta-hires-david-boies-to-seek-damages-from-crowdstrike-microsoft-.html

Re the Delta case - the lawyer they’ve hired successfully sued Microsoft previously on behalf of the US government, and the decision was upheld on appeal too. The ruling almost lead to the breaking up of Microsoft.

The following US government backed out of the case.

Bill Gates said at the time the lawyer was “out to destroy Microsoft”.

So there’s a chance here the CrowdStrike incident may end up having implications across vendor industry around warranties etc, we’ll see.

Jim Cramer does it again.

Replacing an XDR platform at scale takes some time, so if you’re wondering what the translation of Elon’s tweet about Crowdstrike is:

Elon: can we replace Crowdstrike?
Somebody: yes, we’ll begin looking into it but..
Elon: job done

Of course.. given how the Twitter takeover happened maybe he just got them to uninstall it and #yolosec

Delta’s CEO has confirmed they plan to take legal action against CrowdStrike after incurring a $500m loss

6 minute video interview: https://www.cnbc.com/2024/07/31/delta-ceo-crowdstrike-microsoft-outage-cost-the-airline-500-million.html

CrowdStrike shareholders are suing CrowdStrike https://www.bbc.com/news/articles/cy08ljxndr4o

CrowdStrike made a net loss of $845m between 2018 until this year, and has taken on $743m of debt during this period.

Spirit Airlines in the US anticipates a $7.2 million hit to its third-quarter operating income due to operational disruptions caused by the CrowdStrike incident, which forced the carrier to cancel 470 flights.

Here's the Delta boss on his thoughts about the CrowdStrike incident.

They had 40k Windows Server boxes alone, all with BitLocker full disk encryption enabled, all of which wouldn't boot and weren't fixable without manually unlocking BitLocker. That had gone all in with CrowdStrike + Microsoft's most premium offerings.

He has a really good point about how tech companies have become obsessed with growth as their only metric of success, and customer satisfaction is not on the radar.

There's a really mad moment in that interview where they ask them what assistance CrowdStrike have offered, and he essentially says nothing, not even a lunch voucher.

What a time to be alive.

CrowdStrike’s website then vs now

CrowdStrike complained to Cloudflare about a CrowdStrike parody site… and Cloudflare took it down. Without a court order. https://clownstrike.lol/crowdmad/

Cloudflare recently announced they have become a strategic partner with CrowdStrike: https://www.cloudflare.com/en-gb/press-releases/2024/crowdstrike-and-cloudflare-announce-expanded-strategic-partnership-to-secure/

Additionally to loop this in, CrowdStrike submitted a takedown for a parody label (they’ve since rescinded it after being called out).

We’ve reached the part of the brand cycle where people are using CrowdStrike as an excuse https://www.theverge.com/2024/8/2/24212298/mrbeast-beast-games-crowdstrike

360 takes a look at the Crowdstrike kernel drivers - finds they implement an eBPF like system, contain a wide attack surface, don’t check validity of update files (eg no signing of updates) and claim they contain conditions for LPE and RCE vulnerabilities. https://mp.weixin.qq.com/s/uD7mhzyRSX1dTW-TMg4UhQ

Before people write this off as ‘the Chinese’, I’ll give you a hint: there really, really should be security research about the security of security products across all vendors. I’ve seen things.

Previously on Crowdstrike Falcon vulnerability research, check out this timeline where they tried to use NDAs to avoid disclosure, then fixed it without telling anybody. https://modzero.com/modlog/archives/2022/08/22/ridiculous_vulnerability_disclosure_process_with_crowdstrike_falcon_sensor/index.html

EFF are calling for antitrust action after the CrowdStrike incident https://www.eff.org/deeplinks/2024/07/crowdstrike-antitrust-and-digital-monoculture

Bloomberg report a vast majority of the CrowdStrike losses reported by customers will be judged by insurance as not covered by policies. https://www.bloomberg.com/news/articles/2024-08-02/billions-in-damages-from-crowdstrike-outage-to-go-uninsured

I've written up a bit about CrowdStrike's latest bold strategy.

https://doublepulsar.com/crowdstrike-trying-to-use-legal-threats-to-suppress-criticism-and-parody-of-global-it-outage-49320e922120

Microsoft have now queued up to try publicly throw their customer under the bus, claiming (without evidence) Delta’s CrowdStrike woes were due to non-Windows systems. The CrowdStrike issue only impacted Windows systems so I hope somebody at Microsoft knows what they are doing.

https://www.theverge.com/2024/8/6/24214371/microsoft-delta-letter-crowdstrike-response-comments

If anybody wants the subtext of what is happening here, CrowdStrike and Microsoft both really do not want to get sued by Delta and have it go to court as it would potentially be explosive for both orgs and the wider security industry.

The customers are always plebs to be milked, as is status quo.

CrowdStrike incident root cause analysis is out.

Overall, good… but.

It is very verbose but doesn’t say much. Some of the wording will confuse people - eg it talks about rings (waves) in a way which makes you think it is already implemented. It isn’t. They’re saying they plan to implement it later.

Channel updates weren’t tested on a real Windows PC prior to deployment, they relied on automated bespoke code testing. They don’t mention that and it’s the real reason.

https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf

Risky Business take on CrowdStrike root cause report is good.

You can see the confusion the report provides in this discussion I think, eg some of the things are talked about as being implemented - but they’re down as findings for improvement. It’s the way the report is worded, to make you believe certain things existed.. that don’t yet.

https://youtu.be/IcayaFA7OcI

Really good piece about CrowdStrike (technically CSC) misusing DMCA takedown notices over trademark disputes.

CrowdStrike probably want to have a word with CSC about this and Cloudflare should tighten process as DMCA isn’t supposed to be used for this. I know CSC do it.. but they shouldn’t be.

Wider point: cyber industry abusing process in takedowns.

https://arstechnica.com/tech-policy/2024/08/parody-site-clownstrike-refused-to-bow-to-crowdstrikes-bogus-dmca-takedown/

CrowdStrike have responded to two claimed vulnerabilities in CrowdStrike Falcon, including one made by a former staff member: https://www.crowdstrike.com/blog/tech-analysis-addressing-claims-about-falcon-sensor-vulnerability/

There may be more to come on this one..

CrowdStrike vs Delta vs Microsoft continues to play out in public, now with SEC filings

https://www.reuters.com/business/aerospace-defense/delta-air-warns-380-mln-revenue-hit-crowdstrike-outage-2024-08-08/

@GossiTheDog And Delta's CEO asking why this interruption is Windows only shows he should find ways to mitigate this risk. He's shooting in his own foot.
By his own public admission, a responsible company would not / should not be using Windows computers for critical operations.

@kamenrunner @GossiTheDog They are terrified that a US court will rule that the software industry approach to quality of "no care protected by unlimited disclaimer of warranty" will break down. And it should.

@GossiTheDog 1) Certificate pinning will ensure integrity of the delivered content but would do nothing to preserve this integrity of the content once stored on the disk. So, yeah, a MitM attach might not be possible but it's a bit disingenuous to talk about it in the same breadth while "refuting" that the downloaded files can be tampered with on the disk.

@GossiTheDog 2) "A malicious network proxy might block traffic destined for CrowdStrike servers" - no, schmucks, if you respect the proxy settings, a malicious proxy CAN block traffic to and from CrowdStrike's servers.

@GossiTheDog 3) Checksum validation helps ensuring the integrity of the delivered content but is not robust enough to ensure the integrity of the content already delivered, because whoever modifies it on the disk, can modify the checksum too, despite the handwavy "it's stored in a secure location". Like, where? Surely not in the TPM?

@GossiTheDog 4) Yeah, bypassing the ACLs would require that the attacker gains admin rights first - but how many users run with admin rights and isn't the proper question to ask "what harm can the attacker do to and with our product if he gains such rights"?

@GossiTheDog 5) The question of whether the regular expressions that CrowdStrike's product uses are Turing-complete or not is tenuous at best. We'd need to know exactly what kind of pattern matching they use. Standard regexes are indeed not Turing-complete, see

https://www.perlmonks.org/?node_id=809842

But once you get to context-free grammars and higher in the Chomsky hierarchy, things start getting dicer... Anyway, I can't answer this question authoritatively without more information about their searching language.

@GossiTheDog 6) Even assuming a very simple pattern matching language, we have to ask ourselves "what damage can the attacker cause if he can control the pattern?". Is CrowdStrike's product detection-only - or does it try to do any kind of removal/quarantine/etc.?

Surely you wouldn't want the attacker to supply a search pattern that would result in the removal of the contents of the Windows System32 directory?

@GossiTheDog Anyway. My point isn't that their product is vulnerable - my point is that their refutation of some of the points raised by the researchers is tenuous in some places and I would have preferred a better and more clear wording.