Recent searches
Search options
Microsoft quietly snuck out a blog yesterday to say that Office 365 got compromised by China and used to steal emails. Thread follows. https://msrc.microsoft.com/blog/2023/07/microsoft-mitigates-china-based-threat-actor-storm-0558-targeting-of-customer-email/
They used Outlook Web App - runs the Exchange Server codebase btw - to craft tokens to bypass auth.
There's some clever wording in blog around only impacting OWA. OWA is a part of Microsoft 365 and Exchange Online.
The problem was discovered by the US Government and reported to Microsoft. https://edition.cnn.com/2023/07/12/politics/china-based-hackers-us-government-email-intl-hnk/index.html
Microsoft have not linked the blog on @msftsecintel or @msftsecresponse Twitter accounts or social media, instead linking pieces yesterday about an unrelated phishing campaign.
This one looks like a huge mistake, a consumer MSA key (managed end to end by Microsoft - there's no external logs) was able to forge any Azure AD key.
It's only become public it appears as the US Government told Microsoft, which forces public disclosure.
Although MS haven't called this a vulnerability, haven't issued a CVE or used the term zero day.. they don't issue CVEs for cloud services, forging a token is a vulnerability, so it's a zero day.
CISA's advisory on the Microsoft 365 compromise is wayyyyyyyyyyy better than the Microsoft advisory - contains actionable hunting and logging information. Kinda nuts that the US Government are providing better information about Microsoft than Microsoft.
Another element - to spot this activity, the US government used enhanced logging aka Purview Audit (Premium) logging - the US government had a huge public fight with Microsoft over this a few years ago over cost, to get access. Turns out they needed it indeed.
Does anybody have the AppID used in the Microsoft 365 compromise? -> kevin.beaumont@gmail.com
WSJ reporting the Microsoft 365 hack was used to spy on the State Department. https://www.wsj.com/articles/chinese-hackers-spied-on-state-department-13a09f03
For anybody interested - the “acquired Microsoft account (MSA) consumer signing key” used in this must have come from inside Microsoft’s internal network.
The teams who worked on the Microsoft 365 breach of customer data are having a snow day still, I see.
Okay - I found a victim org.
The situation for them is 
MS are going to have to release more info, methinks.. or I crank out the blog writing.
Really good Washington Post piece on the breach of Microsoft 365’s email service.
- hackers accessed customer emails for a month
- Microsoft didn’t notice
- USG had to tell them
- The access to generate tokens very likely came from MS being hacked and not realising
None of these would have helped, since the breach was at Microsoft’s end.
Talked to another impacted victim org in the Microsoft 365 hack, they basically got no actionable info from MS. Basically ‘lol you got hacked’ with wordsmithing and padding. 
I think I’m going to post hunting queries for this with an MS Paint logo.
regulation
I agree with CISA here (and have publicly for years) - security access logs for customers own services shouldn't be locked behind E5 per user licensing.
Yes, it will cost Microsoft money in upsell. They're more profitable than a large portion of the UK economy; they can afford it.
I should also point out the reason Microsoft was able to tell orgs specifically that they'd be targeted even when they didn't have E5 is MS already store the logs anyway.
On how the USG, European govs and Microsoft have been threat hunting the MS 365 breach, per Microsoft documentation on the logs... "If a mailbox is throttled, you can probably assume there was MailItemsAccessed activity that wasn't recorded in the audit logs."
Really good new MS blog on the MS compromise - contains IOCs etc. I'll put MSPaint.exe down. https://www.microsoft.com/en-us/security/blog/2023/07/14/analysis-of-storm-0558-techniques-for-unauthorized-email-access/
“We don’t have any evidence that the actor exploited a 0day." say Microsoft. Their first blog on this says “exploit” - so are MS saying they don’t patch vulnerabilities in their cloud? 
Their latest blog also says “This was made possible by a validation error in Microsoft code” - which is a vulnerability. Which is a 0day as it was under exploitation before Microsoft knew of it existing.
Microsoft lying to media and customers is not a good look.
All it took was Exchange Online in GCC and GCC High getting breached
Non-E5 users to get some security log availability finally.
More details about the Microsoft 365 Exchange Online breach in this article.
Although not stated, orgs are struggling to understand the scope of the breach due to audit log limits on MailItemsAccessed - it stops recording after 1k items. https://www.wsj.com/articles/u-s-ambassador-to-china-hacked-in-china-linked-spying-operation-f03de3e4
Just to loop this thread into this thread - I took a look at the attack path used in the M365 customer data breach.
A key part of the attack chain was documented by Microsoft at BlackHat in 2019.
Wiz have an in-depth look at what they think happened at Microsoft over the Microsoft 365 breach.
They nail a new detail - one of the 'acquired' signing keys expired in 2021, but apparently it was still valid in Microsoft's cloud services. https://www.wiz.io/blog/storm-0558-compromised-microsoft-key-enables-authentication-of-countless-micr
YOU MUST ONLY READ THE OFFICIAL BLOGS
there is no breach
there is no vulnerability
there are no zero days
*jedi wave*
https://therecord.media/microsoft-disputes-report-on-chinese-hacking
The Microsoft write up on how Microsoft 365 got owned to steal customer emails is out. It’s really good and honest from a technical level I think, if you’ve been following the details closely. Top points to the US Gov for forcing public disclosure originally btw.
One big thing missing from Microsoft’s blog (that was in the Wiz blog, and is accurate) - the MSA key expired in 2021. They weren’t checking the validity dates, either - customers might want to ask them if they fixed this.
One extra thing to highlight - Microsoft’s blog doesn’t mention it, but they demo’d the technique of using a signing key to access email from a different account using M365 on stage at BlackHat 3 years ago and made various recommendations to stop it happening again... which weren’t implemented. https://www.youtube.com/watch?v=KN6e1mqcB9s
There’s a pretty good look at unanswered questions the MSRC blog on the Microsoft 365 customer data breach in this: https://arstechnica.com/security/2023/09/hack-of-a-microsoft-corporate-account-led-to-azure-breach-by-chinese-hackers/
Unsurprisingly MS aren’t using words like ‘breach’, ‘vulnerability’ etc when clearly it was both. It’s almost like there’s misaligned incentives.
Other obvious issues include a compromise in 2021 where the threat actor took process dumps etc but nobody checked what they were doing (you live and learn etc), no HSMs etc. Assume MS are compromised.
This TechCrunch piece has one extra detail not in the MSFT blog on the Microsoft 365 data breach - access was gained via session token theft.
To expand, Microsoft use Azure AD MFA, which has a problem with session token theft. https://techcrunch.com/2023/09/08/microsoft-hacker-china-government-storm-0558/
US State Department have gone on the record about how they found the Microsoft 365 data breach.
They set up a detection rule called Big Yellow Taxi two years ago to look for unknown AppIDs in OfficeActivity, which ultimately saved Microsoft’s ass.
https://www.politico.com/news/2023/09/15/digital-tripwire-helped-state-uncover-chinese-hack-00115973
60k emails of the US State Department were stolen from Microsoft 365 in this security breach. https://www.reuters.com/world/us/chinese-hackers-stole-60000-emails-us-state-department-microsoft-hack-senate-2023-09-27/
Microsoft have announced they are going to start using Azure HSM for their own services finally, after being cyber bullied by GossiTheDog. https://www.microsoft.com/en-us/security/blog/2023/11/02/announcing-microsoft-secure-future-initiative-to-advance-security-engineering/
(It’s actually a really good blog with a bunch of good ideas, if you ignore the AI stuff).
Absolutely blistering independent review into Microsoft 365 breach early last year is due this week from Cyber Safety Review Board, highlights huge problems with Microsoft’s security.
I did not participate.
Contains something I didn’t know - last month, Microsoft quietly corrected a blog to say they never found the crash dump with the certificate, so do not know how China got it. They did not store it in a HSM.
References earlier breach they hadn’t disclosed.
Report into MS breach is out: https://www.cisa.gov/sites/default/files/2024-04/CSRB_Review_of_the_Summer_2023_MEO_Intrusion_Final_508c.pdf
I had a tweet in 2021 saying MSTIC should not use the Nation State Notification process to hide breaches from the public.
That was a reference to the Affirmed Networks breach - aka Azure for Operators - listed in this report. They hid it.
The website for Azure for Operators at the time had Satya’s face on it.. that breach, which they refused to share details about, apparently led to this one.
I’ll save full thoughts for later as I need to digest the report, but I will say to Microsoft’s credit, I’ve heard they got the memo on security and plan a range of things including org and governance changes.
IMHO MS need a properly centralised security op model, like you see at.. well.. every other org. And then robust control implementation, lead by risk, blanketed everywhere.
Security should be treated like safety - if you endanger customers, you on the naughty step.
Digging through my old tweets - this one from 2022 was after finding out Affirmed Networks aka Azure for Operators had been breached by STORM-0558 (China).
You will not know about the breach, as it isn't recorded anywhere online other than this tweet. From what I can gather they also failed to tell the US Government about it.
Mindblowing to me that Microsoft had to be repeatedly reminded by essentially the US Government for 6 months to update their own blog to include important information about a security breach... and then nobody even realised they had quietly updated the blog until CSRB pointed it out. Did nobody think through the optics?
The German security services are suing Microsoft over failure to disclose information about one of the Microsoft 365 security breaches: https://www.heise.de/en/news/BSI-verklagt-Microsoft-auf-Herausgabe-von-Informationen-zu-Security-Desaster-9722507.html
I doubt they will get very far as Microsoft takes steps to avoid legal disclosure in security incidents.
@GossiTheDog just read the executive summary. Ouch. More reading on my flight home.
@GossiTheDog When MS announced "Azure For Operators" i just assumed they meant covert agents and the agencies that operate them. With "Affirmed Networks" included in that, I'm even more convinced.
@GossiTheDog ISTR hearing that in 2003.
(Not to snark when they're trying their best and have come a very long way since the Gates memo, but I am weak...)
@BibbleCo @GossiTheDog There was a definite cycle from 2003-2014 (when the TwC org was disbanded). In my experience, that was the high water mark for security at Microsoft -- after that, I felt it became much easier for internal orgs to ignore security for cost, convenience, or a desire to deliver ads on the desktop.
The last major argument that I had at Microsoft, in 2017, was about audit logging in O365. It was disabled by default which led to quite a few companies getting as far as finding a security incident and, when their IR teams went to look for audit logs, coming up blank. The O365 org had the audit logs anyway but would refuse to retrieve them for customers which is some dramatic anti-customer bullshit. I managed to get logs for a number of customers solely because they paid Microsoft for my team's help in investigating their incident which is, with the clarity of hindsight, monetizing anti-customer bullshit.
That's not the argument I was talking about, though. I made the case to O365's CISO that audit logging should be on by default and, while he agreed in principle, he indicated that it would cost about $4mil/month in storage costs. He still agreed to try to make it happen and, after I left Microsoft, they did turn audit logging on by default. I felt pretty accomplished about being part of that.
...only to discover, last year, that they simultaneously locked the most useful audit logging behind E5 licenses, leading to a situation where many customers couldn't even figure out if they'd been compromised.
@neilcar @BibbleCo @GossiTheDog <sigh> I thought for once there was a happy ever after. Seems not. </sigh>
@ancatdubh @BibbleCo @GossiTheDog Well, I, too, am hopeful that being chastised by CISA in 2024 will have some of the same effect on Microsoft that being chastised by Bill Gates did in 2002.
And, in fairness, the problem space has moved a lot in 22 years. In 2002, we were talking purely about product problems -- Windows, Internet Explorer, Office. The solutions started with an embrace of the SDLC and radiated outward from there.
Those problems will never be fully solved and always require vigilance but, on the product front, I think Microsoft does as good a job as anybody in the business. I think Windows is probably more secure, for example, than comparable OSes. But, we've largely moved from software-as-a-product to software-as-a-service and organizations have to wrestle with the security of how they operate and, if they're a service provider, of how their customers operate. I've long joked-not-joked that we need a Secure Operations Lifecycle to go with the Software Development Lifecycle.
@neilcar @ancatdubh @GossiTheDog Very hard to normalise & quantify "secureness" across platforms, even if just considering the base OS, though I've wasted many hours over the decades reading about people trying to do so.
Personally, I continue to use an incredibly obsolete EOL'd system, relying on obscurity, being a low value target, and luck. When I retire from my current occupation (looking after my aging parents), I'll find time to establish a proper home office / labby type environment and get back to the security update fandango, streaming logging to something appropriate, etc. W2K forever! ;)
@neilcar @GossiTheDog "...audit logging in O365 [..] was disabled by default" --
Happily, I never worked at an O365 customer, so didn't know that. As you say, pretty evil. Had I, it wouldn't have violated the law of least astonishment for me... Proper logging should always by on by default, and if the $4m was that big of a deal for MS, they should have rolled it into the basic product cost and spread it across the customer base. Very poor.
@BibbleCo @GossiTheDog Weirdly, I think it was more banal than evil but the difference is often in how we perceive the actor than in the action itself.
And, the sad truth is that O365 is, simultaneously, not great AND the best available hosted e-mail/productivity suite in the market. Maybe Google will apply some Mandiant-sauce to Google Apps but I really wouldn't want to have to manage, detect, and respond in that platform for any large org.
@neilcar @GossiTheDog Concur.
And didn't someone or other once have something to say about banality and evil?
(Godwin? Never heard of it ;) )
@neilcar @BibbleCo @GossiTheDog I'd rather fuck my own asshole with a chainsaw dipped in salt and broken glass than handle an incident in Google environments.
It's laughably pathetic how shit that whole ecosystem is.
@NosirrahSec @BibbleCo @GossiTheDog didn’t know the chainsaw thing was an option.
@neilcar @BibbleCo @GossiTheDog I recently learned of it myself and will gladly preach this from the rooftops until Google decides to delete that whole abomination from existence so I never see it again.
@GossiTheDog Kevin, you are an eternal optimist if you think Microsoft will change in any way for the better.
@GossiTheDog I thought that when Charlie Bell went to Microsoft he was going to try to fix their security architecture amongst other things… EVP Security? He ran a tight ship at AWS. Wonder what happened? https://www.linkedin.com/in/charlie--bell
@adrianco @GossiTheDog I was just thinking that same thought earlier today; cbell@ was legendary before he left Amazon, and it’s been a couple of years now at MSFT and have heard very little.
@adrianco @GossiTheDog I don't know any of the particulars here, but it's not uncommon for people who are superstars in one context to find themselves unable to have the same kind of impact in another context, for a variety of reasons, right?
@jawnsy @GossiTheDog If you are hired as an EVP you get to do a lot of hiring and firing and shaking up of an org. That’s the point.
@adrianco @jawnsy @GossiTheDog even EVP can't cut it if you're swimming upstream against a long established culture.
I've seen CEOs who've swapped out pretty much the entire exec team, and still been unable to make any headway against a 'deep state' praxis (usually centred around finance, HR and legal).
@cstross is right about corps being slow AIs (and people being the gut flora). You can change an awful lot and the stink's still the same.
@cpswan @adrianco @jawnsy @GossiTheDog @cstross hm, so what's the equivalent of a broad-spectrum antibiotic that's strong enough to also require probiotics afterwards to restore beneficial gut flora?
@danhon @adrianco @jawnsy @GossiTheDog I can't think of a single case study where that's been successfully done.
What can work is the equivalent of a brain transplant - merge with another corp and explicitly choose its (better) systems and culture.
Though that can go awry too - viz Boeing post MacDonnell Douglas.
@cpswan @adrianco @jawnsy @GossiTheDog Yeah, I figured as much. I couldn't either.
@danhon @cpswan @jawnsy @GossiTheDog Microsoft when Satya took over is one of the best example I’ve seen quoted as a cultural makeover, open source and cloud first.
@adrianco @danhon @jawnsy @GossiTheDog
Indeed. Though I feel like that's a surface culture, and there's a deep culture beneath that's likely to remain unchanged: what does it take to buy something, or do travel, or spend $ on storing logs, or make the extra effort to add an OpenSSF Scorecard to a repo, (or use you own HSM service)?
There were plenty of IBM CEOs who bragged about changing the culture there, and were lauded in the biz press. But in the long span of history, did they? Really?
@cpswan @jawnsy @GossiTheDog @cstross Agree with that, but still curious about the strategy and progress that Charlie Bell made over the last ~2 years. I skimmed it and he didn’t seem to be named in the report, I didn’t see mention of a systemic improvement program under way.
@cpswan @adrianco @cstross @GossiTheDog @jawnsy last I heard, he spent the 2 years getting buy-in from everybody across the company and building trust. With a proper plan they recently started the actual work on improving things? I could be wrong tho and I haven't validated this!
@adrianco @GossiTheDog to poorly mix analogies, one does not simply turn a $3T cargo ship on a dime. We *do* have incredibly strong security programs throughout the company, but clearly there are gaps that Kev is rightfully skewering us on. The trick is not to fill in those gaps bit by bit, but to build out the program so future gaps fill themselves. Takes time. Lots of it isn't publicly visible.
@SteveSyfuhs @GossiTheDog Clearly that approach hasn’t worked. Need to put some band aids on the open wounds so you don’t bleed to death before you extricate yourself from the war zone.
@adrianco @GossiTheDog I mean we can do multiple things in parallel. The point I was making was that irrespective of things going on right now, there are longer term strategies also brewing. As for the current situation, the messes are getting cleaned up, albeit maybe not at a pace some people expect. Nobody isn't taking this seriously. A year ago we were in a better place than 2 years ago. Today we're in a better place than a year ago.
@SteveSyfuhs @GossiTheDog Good to hear that, but I’m surprised Microsoft is so far behind.
@adrianco @GossiTheDog there's some cherrypicking in that. Applying a broad brush to say all things in the company everywhere are so far behind is unfair. I would say it absolutely applies to some things in the company, but speaking in broad terms doesn't capture the reality well enough.
@adrianco @GossiTheDog also to be clear I'm not trying to defend what happened. I think a lot of failures have happened to get to where we are now and that deserves a lot of scrutiny. I just don't want to see the security folks that have been working their butts off get thrown under the bus because of poor decisions by other people.
@SteveSyfuhs @GossiTheDog Agreed. I’m sure there are a bunch of good people working on this, and I hope things get fixed quicker, but competitor sales teams will pounce on the weakness. We’ll find out over the next year or so how much business impact this has.
@adrianco @GossiTheDog I have no doubt attempts will be made and some will be successful. Unclear how strategic such tactics will be though. There's only a handful that could make a claim of being more secure earnestly, and that runs the risk of flying too close to the sun. Agreed that we can and should do better on a faster time line though.
@adrianco @GossiTheDog the thing is, we had a more centralized security team model back before 2015 or so: it's how I joined MS myself. The problem was that the things that actually were juicy targets like Internet explorer had their own dedicated security teams who did a way better job for what their customers actually needed than the centralized security team was able to do. It's true that most companies don't have differing security teams, but most orgs don't also contain products with such widely varying threat landscapes where you need specialists: the people working on edge's sandboxing stuff probably aren't cloud auth experts we need to stop these recent embarrassments.
I'd like to echo what @SteveSyfuhs is saying about stuff changing: I am very far from Azure/O365 anything but am seeing REDACTED internal fallout of these reports being widespread. 
@SteveSyfuhs @adrianco for the record I agree here. At Charlie’s level, he needs to look at strategy - which takes years to turn around. Rightly so. All the signs are he’s doing a great job I think, because wheels are starting to turn.
Also some cultural change, eg ‘bring out your dead… before attackers do’. There’s lots of very smart people at MS who know about all these problems individually, but organisationally they haven’t been incentivised to say it and fix it IMHO.
@GossiTheDog @SteveSyfuhs The report clearly says that AWS, GCP and Oracle cloud all have far better practices like automated key exchange and reduced key scope and use of HSMs, and I know that AWS has had these for many years. I’d expect the competition to use this to win a bunch of cloud deals from Azure. If I was a CIO I’d be trying to move email (the most locked in cloud service) from Microsoft to Google ASAP.
@adrianco
In my experience of large enterprise cloud sales on the buying side, Deep Dives, careful RFPs & technical due dilligence by experienced practitioners showed Azure to be security scotch tape and bubblegum, but they would win anyway because relationships, pricing, bundling and because it's Microsoft.
IBM disease, v2.
Also MS was very willing to play race to the bottom on price to get the large enterprise deals I was involved with.
@GossiTheDog @SteveSyfuhs
@EricCarroll @GossiTheDog @SteveSyfuhs That’s true until you lose trust. Would you like nation states and Chinese competitors reading your email or not? It makes a difference on the margins, and I wouldn’t be surprised to see a material amount of business impact over the next year.
@adrianco
It's still amazing to me how little execs weight these kind of future & reputational risks. When (not if) the risk realizes, they inevitably are material.
But people focus on the odds, not the stakes, and they rarely pay attention to cumulative risk over time.
@GossiTheDog @SteveSyfuhs
@adrianco @EricCarroll @GossiTheDog @SteveSyfuhs trust: earned in years, lost in moments. Also: “gradually, then suddenly” - and those tipping points are only ever visible in the rear view.
@adrianco @GossiTheDog there's some cherrypicking in that statement. HSMs are already used throughout the environments in most places requiring key storage. Clearly one was not used here and that's a big problem.
@adrianco @GossiTheDog I'm glad you asked that. I was expecting real impact from cbell at Azure.
A 10% of global annual revenue fine for each breach or security issue might also get the message across more directly.
Perhaps the shareholders won't put up with it if their stock dividends are directly affected 
@GossiTheDog Maybe the whole country should think about following schleswig-holstein steps and adotp linux/libreoffice :-)
@portugalense @GossiTheDog Sometimes, I think some form of 5-year enforced prohibition of purchase and use should apply, but with a twist. Mandatory move, and the clock only starts ticking down once everyone has moved; edge cases = the penalty lengthens, encouraging interoperability.