Very big cyber incident playing out at Snowflake, who describe themselves as “AI Data Cloud”. They have a free trial where anybody can sign up and upload data… and they have.
Threat actors have been scraping customer data using a tool called rapeflake, for about a month.
The tl;dr of the Snowflake thing is mass scraping has been happening, but nobody noticed.. and they're pointing at customers for having poor credentials. It appears a lot of data has gone walkies from a bunch of orgs.
Snowflake is a big AI data company with a conference in the US next week, chances of that going ahead are interesting.
IOCs: https://community.snowflake.com/s/article/Communication-ID-0108977-Additional-Information
Snowflake admin users need to check their Snowflake environment, not sec departments check their on prem.
Five orgs have told me they are running incidents for Snowflake, where their data has been copied.
Snowflake: there is absolutely no cybersecurity incident.
Also Snowflake: Please run these commands and look for "threat activity" logins with the user agent "rapeflake" using this knowledge base article we haven't listed on our website.
https://community.snowflake.com/s/article/Communication-ID-0108977-Additional-Information
Live Nation said its stolen database was hosted on Snowflake, a cloud storage and analytics company.
I've now confirmed 6 major orgs running Snowflake cyber incidents, so I've made a theme song about Snowflake's response.
The deleted Hudson Rock post on Snowflake breach: https://web.archive.org/web/20240531140540/https://hudsonrock.com/blog/snowflake-massive-breach-access-through-infostealer-infection
For the record I don't think all the content is accurate - however Snowflake did have a security incident via their former employee, they have full IR stood up. They didn't follow their own best practices.
I also know multiple orgs who've had their full databases taken from Snowflake.
I wrote a blog on everything I know about the Snowflake situation https://doublepulsar.com/snowflake-at-central-of-worlds-largest-data-breach-939fc400912e
The Snowflake authentication setup is terrible.
MFA can’t be enabled org wide, each user has to manually log in and enable it. There’s no policy to block users without MFA. And it uses Duo MFA rather than your orgs MFA. (You can bring your own MFA with SAML).
Also all users log in via a Snowflake domain, so you can just pull creds from info stealer marketplaces or logs.
That’s why they’re being targeted as a platform.
Hudson Rock have put out a statement saying a legal threat from Snowflake caused them to remove their blog. https://www.linkedin.com/posts/hudson-rock_activity-7203433945919578113-RH05 HT @mattburgess
Another Snowflake customer breach: https://techcrunch.com/2024/06/07/snowflake-ticketmaster-lendingtree-customer-data-breach/
One thing I didn't know until recently is Snowflake has a massive fanbase, Apple and Amiga style - if you critique Snowflake in any way people flip tables. The comments on my blog are fun. I mean, the clue is in the product name, really.
IMHO it's fair to call out Snowflake's authentication isn't very good - it's the worst SaaS MFA solution I've seen as it has no top level, easy switch for org wide MFA enforcement.
Combined with putting all customers under *.snowflakecomputing.com sub domain is why their customers are getting owned - infostealers are just full of creds ready to go.
I gather Snowflake are discussing changes to fix, don't tell the fanboys (and yes, they're all dudes).
Mandiant have informed 165 organisations they may have had data exfiltration from their Snowflake hosted databases
https://cloud.google.com/blog/topics/threat-intelligence/unc5537-snowflake-data-theft-extortion/
Kinda interesting - Mandiant notified Snowflake that over 100 customers had data exfil issues, and Snowflake’s share price immediately began to tank in sells offs - before the incident was made public.
won a game of Call of Duty
hacked the world’s largest companies
used an infostealer
Can’t wait for these guys to have super secure Microsoft Recall, which is definitely encrypted from the user
Snowflake have told customers "We are also developing a plan to require our customers to implement advanced security controls, like multi-factor authentication (MFA) or network policies, especially for privileged Snowflake customer accounts."
Good! They also say the attack was "not caused by a vulnerability, misconfiguration, or breach of its product". Just happy little bad MFA.
Nice: "In a phone call this week, Jones (Snowflake CISO) told WIRED that Snowflake is working on giving its customers the ability to mandate that users of their accounts employ multifactor authentication going forward, “and then we’ll be looking in the future to [make the] default MFA,” he says."
This will be a great outcome for Snowflake customers and Snowflake itself. I know Snowflake got big mad at me for pointing it out, but that was a prime weakness in their MFA.
https://www.wired.com/story/epam-snowflake-ticketmaster-breach-shinyhunters/
The Hudson Rock blog earlier in this thread on Internet Archive has also been removed. https://web.archive.org/web/20240531140540/https://hudsonrock.com/blog/snowflake-massive-breach-access-through-infostealer-infection
The Hudson Rock blog on Snowflake, archived elsewhere: https://archive.ph/2024.06.01-023241/https://www.hudsonrock.com/blog/snowflake-massive-breach-access-through-infostealer-infection
65 page PDF on searching for Snowflake malicious activity: https://services.google.com/fh/files/misc/snowflake-threat-hunting-guide.pdf
When Snowflake allows orgs to easily mandate MFA across their users, I plan to answer this forum post from 2019. https://community.snowflake.com/s/question/0D50Z00008ugjwISAQ/is-there-a-way-to-force-all-users-to-use-mfa
Cisco has a look across the wider infostealer problem, using Snowflake as a jumping off point: https://blog.talosintelligence.com/infostealer-landscape-facilitates-breaches/
I think SaaS providers who provide their own authentication have a responsibility to provide robust, *enforceable* MFA for their customers - so if an org wants all their users to require MFA, they can and it’s just an easy tick box.
Some SaaS providers aren’t doing this - - and it’s the reason infostealer logs are such a problem. Their angle is customer is solely responsible, but as a counterpoint: see how that is working out for Snowflake.
Snowflake have rolled out MFA changes:
- A new authentication policy that requires MFA for all users in a Snowflake account
- prompting for user-level MFA setup
- Snowflake Trust Center for monitoring adherence to MFA policies
This solves all the inherent product weaknesses from the prior setup, they did a good job.
https://www.snowflake.com/blog/snowflake-admins-enforce-mandatory-mfa/
AT&T become latest org caught up in Snowflake incident - they’ve had phone call records and text messages stolen for nearly all customers. https://www.404media.co/hackers-steal-text-and-call-records-of-nearly-all-at-t-customers/
The person who got arrested for this is likely a kid btw, there’s a bunch on Telegram who operate with terrible OPSEC.
Turns out yeeting your most sensitive data into a “AI Data Cloud” with no enforceable (at the time) MFA was a bad idea.
@GossiTheDog I don't think that's completely accurate. I login to a couple of Snowflake accounts with my organisational SSO which includes our standard MFA.
@GossiTheDog Our (fairly large) org uses SAML federation, with MFA enforced.
@GossiTheDog @mattburgess Snowflake by name…
@GossiTheDog @mattburgess Oooooh shit.
huh. that's a real old school IBM move.
what do you do when you have better lawyers than tech/security folks? sue for defamation, patent infringement, and anything else you can think of to try to quiet competitors or embarassment.
@GossiTheDog @mattburgess That's good considering Hudson Rock's post was completely wrong. Not to mention calling out a specific employee that they believed had been compromised is not only unprofessional, but completely inappropriate. If it had been true, can you imagine the psychological toll of knowing that you were inadvertently responsible for something like that?
@GossiTheDog Couldn't possibly be related... /s
@GossiTheDog maybe it was a bad idea to just dump all the data into one big pile with just a big sign saying “bad guys stay out” defending it.
@GossiTheDog so has their summit actually been going on for the last 3 days? Not sure I'm willing to sign into the hellsite to look for the hashtag.
@GossiTheDog Similar to the old hardware reverse engineering practices where the people touching the original hardware and the engineers designing the clone are never the same people, I wonder if there are quiet companies out there who buy stuff like this and then sell a “customer lead db” to competitor stores in a way where the competitor can legally pretend it isn’t stolen data.
> putting all customers under *.snowflakecomputing.com sub domain
How else would you have them handle it? Nearly all SaaS providers do exactly this.
A popular one you might have come across is *.github.io.
(Not a fanboy; I just work for another SaaS vendor and would like to know if we’re doing anything terribly wrong.)
@GossiTheDog Yeah we were setting up MFA for a Snowflake SSO break-glass account and the enrollment process didn't work ("success" but then locked out), had 2 more people try and it didn't work for them either (same issue), this was months ago so it's probably fixed but I wouldn't be surprised if enrollment friction played a role
@GossiTheDog
@Johnhultquist
Even if Snowflake had "proper" MFA authentication across the board, would that have significantly changed the outcome?
That is, if an info stealer is what's at play, and it steals my authentication cookie, I can't really rest at night knowing that MFA is keeping me safe, right?
@wdormann @GossiTheDog @Johnhultquist I think this is why we need device bound credentials, but that spec is still in progress. MFA at least helps with people brute forcing passwords, but rate limiting would help for that too
@jawnsy @wdormann @GossiTheDog @Johnhultquist
How do you device bound credentials? I have only heard of IP bounding them (so if it’s suddenly being used from a different location a new login is required)
And that is already being used by some companies afaik.
@Euph0r14 @wdormann @GossiTheDog @Johnhultquist I don't think you can until this (or something like it) is added to the web standards https://github.com/WICG/dbsc
@GossiTheDog that seems like a lot
@GossiTheDog some of their telemetry data has and it doesn’t contain customer info…
@GossiTheDog SEC be like
@GossiTheDog people who still use #Windows11 should be legally barred from doing anything re: #ITsec.
@GossiTheDog seems like misconfiguration if it can be fixed by forcing customers into using a different configuration…
but I guess it wasn’t a misconfiguration of snowflake’s own production account(s) at least
@GossiTheDog Oh my "It's a happy little cyber incident!" is now stuck in my head.
@GossiTheDog into the memory hole it goes
@GossiTheDog out of the memory hole it returns, lol
@GossiTheDog Do we know why it got removed? Did Snowflake C&D them? And if so, why, is there anything manifestly untrue in it? (Looked to me from the archive that HR was careful to state that this was all 'claimed by the hacker', not proven facts).
@GossiTheDog chefskiss.mp4
@GossiTheDog "I would also like to have a setting that enforces MFA. Hopefully this can be implemented soon," said Bas... *two years ago*.
@zackwhittaker @GossiTheDog Exactly.
Blaming the users on their password hygiene, when you don't offer org admins any way to enforce SSO or MFA, is a bit ... let's say, not a great look
@GossiTheDog if the SaaS vendors could provide SSO integration without having to pay extra, it will definitely help reduce these type of events. Stop putting paywalls on basic security features. Not all companies can afford to pay for enterprise licences to get SSO.
@GossiTheDog @wrw I see a lot of SaaS offerings that absolutely jazz hands about shared responsibility more often than not, and we can hitch most of them to our azure SSO. it's usually fine!
but occasionally we see some weird approaches. if a vendor can't integrate with an IDP and they're managing credentials i agree they should provide MFA and honestly why not require it? if you're dead set on not using it, use your own directory and not mandate MFA yourself but clearly your own bad decisions.
@GossiTheDog It’s a good start! #AWS doesn’t even let admins force #IAM users to use #MFA !
@GossiTheDog all it took was a massive data breach that affected millions of people
@GossiTheDog I don't get how this isn't default for cloud services. Better late than never I guess.
@GossiTheDog The word Snowflake just gives me the mental image of someone taking this post out of context and turning it into a conspiracy theory surrounding either multi-factor authentication or, alternatively, grad school writing workshops.
@GossiTheDog (adds a nick to an already full scoreboard titled "proof that keeping customers' records is a bad idea")
Senator Ron Wyden told 404 Media in a statement that “This is not the first data breach revealed by a major phone company and it won't be the last. These hacks, which are almost always the result of inadequate cybersecurity, won't end until the FCC starts holding the carriers accountable for their negligence. These companies will keep shortchanging customer security until it hits them in the wallet with billion dollar fines.”
Every time I see a quote by this guy, it's fucking gold.
@GossiTheDog Notorious hacker known as “Snowflake” strikes again.