Posted by galadran 11 hours ago
"Ten days later, macOS and all applications that rely on it still trust the compromised eMudhra certificates" would have been a better title.
Apple fucked this up, but I really don't think the situation is "as bad if not worse" than if MacOS just didn't do this.
There’s very little actually running in a docker container. ca-certs may actually be an exceptional case though, since it’s not code that’s running but a kind of data.
It’s not like if you use an Ubuntu image for a node.js app that you get PHP too and have the PHP vulnerability present in your web app.
It's same problem we have when libraries get massive security vulnerabilities. Updating them is not always as easy as many would like.
That's the world we live in now thanks to Google and Mozilla.
But this abstract exercise is only to highlight the importance of Apple getting its shit together, because such a scenario exists with Windows and the culture of downloading exes and msis and it's honestly way worse than having a central but responsible authority.
Is it possible for users to remove the eMudhra certificates themselves? I have no experience with MacOS, so I don't know if certain actions are prohibited. To be clear: I am not suggesting that normies do this! Rather, I am asking if it is possible for highly skilled users.
Certificate management is fifty kinds of all messed up - and in THEORY everything is designed perfectly and properly, in practice it's all bandaids on bandaids that nobody cares about as long as "things mostly work".
So from a spherical cow perspective, it's well-designed; from a "this has to work in the real world with real things and people" it's ... workable by violation of the design all the time.
Tangentially related: I fucking hate so much that browsers now all have their own DNS resolvers and caches, it's just Another Fucking Thing to debug when you're working especially on locally hosted stuff. This is not a browser's job, it's a task that should fall to the OS.
Welcome to hell. https://en.wikipedia.org/wiki/DNS_over_HTTPS
(My current chrome reports no at https://one.one.one.one/help/ so maybe I changed it or maybe the default's different now)
Edit: Found it, February 2020: https://blog.mozilla.org/en/firefox/firefox-continues-push-t...
Out of curiosity, what are some of those things it messed up for you? And what kinds of builds is it affecting?
And like, I don't need that. My DNS server is properly maintained and queries those exact same servers itself when you request something like google.com or apple.com. Putting a separate friggin DNS query AND cache in a browser is just more trying to protect people from themselves IMO, and introduces this weird issue where you can't hit an internal site in Firefox, even though you're running `nslookup` on your freaking computer, and it's giving you the proper IP address. And paradoxically putting the IP address in makes the browser find it when the domain you can see correctly resolving in terminal isn't resolving in the browser is fucking obnoxious.
It also would render a pihole useless because Firefox despite the PC's DNS not resolving ad provider's names, WILL resolve them, ensuring an attack vector for malware remains wide open even if you've tried to close it.
And while I'm bitching about this: STOP DEFAULTING EVERY CONNECTION TO HTTPS. Any properly configured website is going to do this anyway, the browser also doing it, again, just annoys the piss out of anyone trying to host stuff locally because why the fuck would I get certificates for servers that aren't even exposed to the damn Internet!?
Edit: Git is still affected, we're fixing that now.
(Source: I am one of Homebrew’s maintainers.)
There's a fix incoming here[1].
> Homebrew ships its own CA bundle
Outside of an enterprise environment, I never thought about having multiple "golden sources" of CA certs on a single host. (I have no experience working outside of classic enterprise environments, so please do not read my surprise as immediate dismissal of this idea!) For those unaware, in enterprise environments, it is very common to use a common CA cert pack from Oracle/Mozilla/Google/Microsoft/Amazon/Apple, but add 2-5 CA certs from your own company that act as trust roots.
I did some Googling on this matter. It looks like the "Big Four" of CA cert packs are: Mozilla, Microsoft, Apple, and Google. (I guess there is one or two in China also, be leave that alone for the moment.) What happens if they are in disagreement? This is a complex problem! If anyone is involved in this process, I would like to learn more.
Not easily, because it's closer to a convergence of smaller engineering decisions into an emergent choice to ship a CA bundle.
But the short version: lots of tools ship their own CA bundle. This is not ideal for distributors like Homebrew, because people expect one CA root program for the entire distribution. As a result, Homebrew tries (and sometimes fails) to apply a consistent root program across its tools. Apple's own roots are not inherently best suited for all Homebrew users, so we end up shipping a composite of their roots and the Mozila root program.
> What happens if they are in disagreement?
They are in disagreement, all the time! Root program operators are totally free to manage their trust roots as they please. The only forcing function is a desire to honor users' connections; differences in root programs end up getting papered over somewhat by cross-signing between different CAs.
Edit 2: This is being fixed now[1]. Everything below is stale.
I couldn't reproduce this:
% which git
/opt/homebrew/bin/git
% git ls-remote https://git-emudhra.we-have-a-problem.com/check.git
fatal: unable to access 'https://git-emudhra.we-have-a-problem.com/check.git/': SSL certificate problem: unable to get local issuer certificate
$ git ls-remote https://git-emudhra.we-have-a-problem.com/check.git && which git
2975b9930644b321e18ecae09e29c15b955e6103 HEAD
2975b9930644b321e18ecae09e29c15b955e6103 refs/heads/main
/opt/homebrew/bin/git
Wow, that was fast!
I would've preferred a doc or a code repo link as a source. Otherwise, this just reads like a brag. A decent one, to be fair :)
(Sorry if you thought it was a brag, it’s just supposed to be context so that everybody is on the same page about the reason for a comment.)
I understand why they originally let each CA in, but it's definitely not one size fits all users.
A user in the US does not need to trust a Turkish CA the same way a user in Turkey does.
There needs to be an easy way for end users to personally drop trust of CAs they don't need and have that decision respected and remembered.
There is. On macOS go to Keychain Access > System Roots and disable the kind of trust you want to disable. There's probably a scriptable method for that.
On Windows, open the management console to "Certificates". Then go through the list and move/remove/distrust any certificates you dislike. There are also powershell tools you can use.
On Linux, the situation is a bit annoying. You can remove all CA certificates manually and make the directory read-only to break any attempt from your distro to interfere with the list. You need to check your distro documentation to see if they use /etc/ca-certificates or /etc/ssl or something like /etc/pki for reference. Some distros offer a partial GUI as well.
Programs that deem themselves smarter than the OS (true in the case of macOS and various Linux distros), like browsers, the JVM or NodeJS applications, can have their own cert stores.
You will need to acknowledge half a dozen certificate prompts for every website you visit after this, though. Make sure to compare the fingerprints of the CA root certificate with the real ones you might find elsewhere, because it's easy to generate your own CA and copy all certificate information signed by your self-generated key.
- Only ISRG(Let's encrypt) being the sole CA and benevolent dictator of the internet. (then forced to be dissolved due to antitrust?)
- Each country controls their own CA and you need to purchase 200+ certs in order to make your website generally accessible. (then country XXX will charge you $200/year)
- Big techs owning OS/Browsers control their own CA and cross license each other (then Apple still being lazy handling incidents) (then they form a cartel and block others, like current Tier 1 ISP, email senders, credit card networks)
edit: put enter between numbered items.
So someone has to do it, and a pop-up on first boot asking you which "certificate providers" you want to trust would be as pointless as the "which browser do you want to use" ones were.
If I can speculate though on why these bundles come from the browser or OS and aren’t fully in the user’s responsibility. You’d have to start from scratch for it to not have a bunch of stuff the user had ‘no need for’ to be present, and the user would need to see a cert window semi-frequently that gives the possibility to start trusting a CA, and the bottom 99.99% of users in sophistication has no idea whether they should say yes or no. I mean, without being signed by a root CA you already trust, even the name on the new CA could be completely made up. “Do you want to trust the ‘Apple Root Certificate Authority’?”
But even if not, you get asked whether to turn on Siri, whether to send telemetry, whether to sign away your rights, etc., all of which are consequential but prompts are massaged in a way where it's nice and easy for the user to answer a certain way without having a clue. Makes me think user comprehension isn't even a problem.
Where I suspect the problem lies is that it will not fly to make it seem like US-based CA's are more trustworthy just because the majority of sites roll up to them. Then you'll need to do some non-country-specific human tiering, and who's the judge of that?
So what happens to americans visiting french sites?
[Do Not Trust], [Trust Once], [Trust Forever]"
The country comparison doesn't do much either, because there isn't a good guarantee that they'll correspond. Some guy posted a repo to distrust every CA except the top 30, and that has a bunch of weird examples, like a bunch of russian sites being signed by a Greek CA. Not to mention some of the biggest sites use CAs that don't correspond with their country of origin. GlobalSign is Belgian for instance. Guess how many of their customers aren't Belgian?
You can add other options too, "Trust for 30 days" or something. I wish SSH known hosts has an auto-expunge.
Thanks for being on the ball, Apple.
I don't know why they haven't rolled out an update if they've already wrote a whole article about patching the problem yourself.
But instead the Apple fanboys will be out in force to tell us why actually its a good thing for Apple to leave parts of the OS insecure.
It really surprised me what mobile devs put up with, just an utterly horrible hostile experience.
Better luck next time.
I have no idea how you've come to this conclusion, and clearly nothing of the sort is actually going on here.
If Apple is still trying to brand themselves as security-forward, you'd think removing the revoked CA certs from /etc in a system update would be a pretty easy thing to do?
For your tool, remember to nuke Windows' automated certificate authority updates (separate from normal Windows updates) and possibly the automated intermediary cache. Microsoft publishes a list of trusted CAs/intermediaries periodically and they get automatically applied to Windows so websites don't need to send the full cert chain during TLS negotiation, and it has a tendency to cache trusted intermediaries as well.
Mozilla's trust store is one of the first to get updated when a revocation happens, as well as Google's CRLsets. Downloading full CRLs repeatedly would be a waste of time and bandwidth, that's why there's a specific distribution method for CA updates. I would extend your tool with a daemon that checks for revocation automatically, so you don't accidentally eMudhra yourself while browsers and the OS get updates next time something like this happens.
This feels like all of the C code getting attacked now where everyone spent the last three decades building up cruft because fixing it would be a chore, and now the separate bills are coming due all at the same time.
Then someone would ask "there has to be a better way" again, when the reason for the new inconvenience gets forgotten, and it oscillates.
Fundamentally, people don't want to think about the intricacies of how we decide whether or not to trust a particular cert. They want it to just work without thinking about it. This includes that if they have a custom private root they need to trust, they just drop it various places until it starts working and call it done. And sometimes they do weird things like adding intermediates to their trust store because they were trying to call a service that has their cert chain configured incorrectly, and that "fixes it". Again, it's a nightmare.
The fact that linux, a system notoriously cobbled together by 873 different factions on the internet, is even more of a mess is not surprising in the least.
Not great for unaware developers, but most macOS users are fine. Those who are aware can follow the steps in https://support.apple.com/en-us/149555 until Apple bothers to roll out an update.
I suspect the fast-track background security updates can only update the databases and revocation lists that live in the proprietary APIs, not any arbitrary file on the system
I don’t think it’s gross incompetence, since they documented the steps to remove. They may have some unexpected dependency on the file? Looks mysterious from the outside.
Dropping the Sectigo root along with the eMudhra root is the easy and safe solution, and maybe this is what Apple should have done (it’s the workaround they recommend!). But it means that some small percentage of certificates that are fine won’t be accepted.
Correct, those usually use Apple's networking APIs which all go through their security stack, which already picked up the OCSP revocation within about 2 hours in most cases.
my guess is their thinking is that if you're running those apps on Mac, you're a developer or know what you're doing, and can edit /etc/ssl/cert.pem until they push an update
they also have to consider that such users may have custom changes to their cert.pem file, and they probably don't want to rush an update that would potentially fu-bar those changes
not convinced? take a look at your /etc/ssl/cert.pem
Pretty much every developer on mac should probably run their patch.
Per https://emudhra.we-have-a-problem.com/apple.html
> macOS 27.2, released on 26th October, did not fix the issue and undid the workaround that users had applied by hand, leaving them vulnerable again.
Apple, for the company that prizes itself on believing privacy is a human right, why are you failing your customers so badly on PKI management?
It’s completely insane to me that Apple didn’t script this with interactive prompts as needed, and non-interactive options for automatic installs.
On Mac, some command-line tools included with macOS or the Xcode Command Line Tools (including curl, git, python3, ruby, perl, and openssl) check certificates against /etc/ssl/cert.pem. For certificates that chain to a root CA certificate in this file, these tools don't check Apple's block or whether the certificate has been revoked. This file will be updated in a future macOS update. If you use these tools and don't want to wait for the update, you can remove root CA certificates from the file.Tools common on Linux like curl and git come with the OS basically for developer convenience. Apple doesn't write those programs, they don't use Apple's APIs or platform features, and thus in some sense you aren't really meant to use them in Apple's eyes. Ports of UNIX CLI tools exist in a kind of semi-supported netherworld on macOS. They run but they aren't fully integrated or supported parts of the OS.
This is technically a problem in the CLI - it needs to support revocations properly, not rely on root CA verification only. I’m not sure though whether there’s a standardised mechanism (that Apple supports) for CLIs.
> These steps affect every tool that uses /etc/ssl/cert.pem. Afterward, tools that use only this file might not be able to verify some websites whose certificates chain to these root CA certificates through intermediate CAs that weren't affected.
Not sure why macOS doesn’t ship such a build of curl.
So the only way to fix apps that don’t handle revocation of intermediates (eg the ones that only look at certs.pem) is to revoke the root.
Great, that’s fine. They can reissue.
The writing is on the wall for the command line tools.
(We can already see this tooling devaluation happening all over the place. Apple have their variant of this response to the AI tooling-paradigm tsunami…)
I’m not saying they’re going to deprecate XCode today, but I do believe they will “agent’ize” Developer Tooling drastically in the next year or so. Agents are becoming the most important user of command line tools.
Because they have to, in order to catch up with NVidia, their only real competition. If you want to understand why I think that, its because I have been developing and deploying cross-platform Linux-centric (but MacOS and Windows targeting) applications, mostly for the ARM platform but also with Intel (SIMD on both) .. for the past few months, and I haven’t touched an IDE in a month. On DGX Spark. Which turns out is a truly, utterly, mind-blowingly great platform to do cross platform high performance application development…
CLI is just a thing that NeXT had and even that made it easy to use GUI tools.
Xcode is a GUI tool – it happens to expose some cli tools but they could be covered up – however some developers would complain a lot so I don't see that happeneing.
The original question is, why aren’t Apple fixing the CA problems for the command line tools (whereas they have done it for .app’s such as Safari), and the reasoning is, because the user will just have to fix it themselves, because Apple don’t care about the command line users - they see that these command line tools are, indeed, becoming the front-line for agentic development, so .. presumably, they have an .app bundle which does away with this problem, to be shipped on the horizon ...
I wonder why they stopped doing that and now just seem to do "standard" builds and end up with this (at least) 2 separate trust store situation.
They might need to make a way to do this outside of os updates
https://fullmetalmac.com/cybersecurity/macos-security/xprote...
You're either not an Apple user, or never noticed Settings → General → Software Update → Automatic Updates
https://support.apple.com/en-us/102657
The big question is why they’re not using it for things like this, where a tiny layered file system update would benefit almost all of their users immediately.
I have no evidence or proof, but I am convinced this wrap/unwrap key generation debacle was intentional and the weak keys were discovered and kept secret by the agencies or governments and used to exploit their adversaries.
You don't need to perform a traditional MITM attack _if you own the airport_ or you _own the free public wifi_. The traffic can be harvested in clear text and we have to come to terms with the _remote possibility_ that this was going on for over a year and a half. If not that long, then certainly for the past 10 days.
So, I do agree this could be problematic, on specific cases where people not using a major browser were accessing resources some is interested on, on a network they have control of. Makes the surface quite small.
The threat here is more so for 'backend' services than regular browser users.
Editing to add a link to this Apple support article which is unsurprisingly detailed - https://support.apple.com/en-us/149555
Similar reason (though with the opposite effect) as how a cert can be fine for browsers but fail for curl when they don't send an intermediate: the "vanilla"/library/command-line stuff is just checking for a path to a trusted cert and not doing the CRL or AIA stuff a browser does.
CLI-based activity references etc/ssl/cert.pem. While they say this will be updated in the future (likely the next dot release), admins can also proactively remove the eMudhra entries.
macOS already has the verifier needed (Security.framework/trustd path that protected Safari). It would be good if MacOS routed CLIs through it.
Why does that matter in this case? Can't this be fully mitigated with next to no collateral damage just by removing the trusted roots that signed them?
Dropping the Sectigo root along with the eMudhra root is the easy and safe solution, but it means that some small percentage of certificates that are fine won’t be accepted.
If I’m an OS vendor I would have been putting them on notice days ago that their root was being dropped and they have X days to reissue certs and deliver a new root cert or otherwise they can just accept that they’re not qualified to be a root anymore.
# /usr/bin/curl https://dv-g3b-aaa-noct-emudhra.we-have-a-problem.com/
Your client successfully connected to the site. That's bad!
# sudo mv /etc/ssl/cert.pem /tmp/
# sudo ln -s /opt/local/libexec/openssl3/etc/openssl/cert.pem /etc/ssl/cert.pem
# /usr/bin/curl https://dv-g3b-aaa-noct-emudhra.we-have-a-problem.com/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.htmlyeah, it's not ideal but the scale of the threat is way overblown tbh. You have to compromise a lot to be able to make a MITM here.
Let's not treat it like PHP2Shell.
It may not be directly exploitable on its own, which is the impression some reports give, but it is a serious hole to leave open especially with the current rate of old security issues being found, as it could be _incredibly_ useful to a malicious actor as part of an exploit chain.
xz-tools was only caught because it had a bug causing slow startups and someone was micro-benchmarking Postgres and decided to dive in further.
I agree it's not PHP2Shell, but its certainly not being "way overblown" here.
Users will be none-the-wiser because they're not accessing these hosts through a web browser, and you net a few more compromised hosts in your inventory.
Edit: Holy moly we all jumped on this comment at the same time didn't we.
It's good to see push back against this narrative that is trying to develop that this isn't any kind of big deal.
This has been every machine on the internet compromised for at least 12 hours and probably some that are not going to ever get patched.
This is a HUGE deal for embedded devices, we can now mitm HTTPS traffic on lots of embedded devices running Linux produced between 2018 and now.
The entire reason one can rest easy as they connect to their coffee shop's WPA2-PSK network, is the security provided by TLS.
And if you don't think an ISP or agency would abuse this if it could be done undetectably then you haven't been paying attention these last 20 years (hence why DoH is now a thing).
Either way, I mean rouge in the sense of an AP that uses the same SSID and password as the AP of a coffee shop or office to have devices automatically connect to it without their users noticing.
“Rogue” is a completely different word. Rogue access point.
So this isn't like Firesheep [0]. Yes, there would be vulnerable traffic, like the git client this thread is discusing, but it's not "everything"
This is just an argument against TLS in general. Use plain http if you believe this.
I’d also bet more things use libcurl behind the scenes than we appreciate, and that at least a few of them have things like silent updates which retry after a failure without notifying a user. I’ve seen a lot of agents which just try to periodically check in because they’re designed not to be noisy on lousy networks.
It's trivially easy to mount an ARP spoofing attack. I did that at school two decades ago and on-the-fly rewrote <img src=...> to hardcore porn images for funsies.
mitmproxy [1], a popular toolkit for making that easy, has the option to generate "eMudhra" certificates on the fly now.
It is trivially easy to compromise things, all you need is physical access to the network.
If you get past all of that, you still need a victim who is actively using a cli tool on macOS using https.
It’s not impossible, but the conditions are much more specific than those needed to inject porn into a webpage on a poorly configured school network in 2006.
So you need to somehow trick macOS into remotely connecting to the wrong bssid, and then arp poison a user who happens to be using a macOS cli tool that’s using https.
Sounds pretty targeted and unlikely to happen.
I don’t think that’s either targeted or unlikely?
I just set up my phone's hotspot to be the same SSID and password as the network the robots already know from the office, and I'll have the robots, a couple phones, and a laptop connected in 10 seconds.
Yeah that's not so much "trivially easy"
I hazard that the answer to all three of those is "yes".
yeah, it's trivial to punch somebody in the face in the morning, all you need to do is to be by their bad in the morning when they are waking up.
If you think it's not that important, it becomes a critic about TLS and basically saying that ciphers for web traffic are useless.
Physical access to networks isn't that hard to come by.
You'd be better off just mugging them in the parking lot.
Except it won't impact gmail unless you're accessing it through curl. The browsers already distrusted the compromised intermediates via CRL within hours of disclosure, and removed the affected root via update.
What if it's the Starbucks at 1730 Pennsylvania Ave NW, Washington, DC 20006 for an extended period of time? Compounded with other security issues like password reuse...
We can always draw a line and say, "that's not practical". The assessment an attacker is likely to do is going to be "is this practical inside my budget for the value I hope to acquire?" or maybe even just "waow, lulz".
Regardless, real bad actors are busy compromising your next `npm install` anyway. Much better reward.
If someone has it in for you and hired a high skill attacker, its not a stretch to imagine they compromised your home network.
Why are you assuming there isn’t someone else trying to hack you at the next cafe? Actual damage can happen one person at a time, and can occasionally lead to larger scale compromises. Randomizing your location isn’t going to reduce your risk. Now that everyone who’s paying attention knows they can MITM your Mac, you shouldn’t trust any network.