Posted by mobeigi 2 days ago
Hopefully a wake-up call to those who believe older distro LTS kernels are getting all the security fixes Canonical and Redhat would want you to believe.
> One of the first things our security team did was confirm that our existing endpoint detection would catch this exploit. Our servers run behavioral detection that continuously monitors process execution patterns. It doesn't rely on knowing about specific vulnerabilities; it watches for anomalous behavior across the fleet.
I have a fileless, execless copyfail exploit that works by injecting shellcode directly into systemd's pid 1. (I should probably publish it at some point...)
There's no reason the task should even be assumed to be executing code in a file. A process can map code into anonymous memory and continue executing there without even branching. Again this is considered a feature of the system rather than a flaw.
In a serious environment you'd run IPE with dm-verity/fs-verity to ensure binaries are whitelisted and integrity-checked at every execution.
CopyFail only highlights why Companies want LTS. If there was a supported kernel built prior to 2017, most large companies would still be on that version, avoiding this issue all-together.
The corporate mindset is usually "never upgrade unless there is new hardware needed or critical software failure". All CopyFail did was reinforce that mindset.
I wonder if CopyFail will cause enterprises put pressure on the Linux Foundation to maintain a "ultra LTS" were it is supported for 20 years ?
Sadly not really how it works for say Red Hat. They routinely backport features while keeping whatever "stable" number on kernel. We even had displeasure of them backporting a bug... same bug to 2 different RHEL versions
* Get list of modules from Puppet's facts, confirm module isn't used anywhere (it wasn't) * `install algif_aead /bin/false` in /etc/modprobe.d/disable-algif.conf * Run a check using exploit code to check it is no longer working
I imagine CF runs more stuff that could use it I guess but apparently it's not often used API
Although given the tendency for end point logging agents to run on buffers to reduce their network chattiness I do wonder if a fast acting exploit could dump that buffer before it manages to be transmitted.
I don't think any of the agents are complex enough to immediately transmit permission elevation log messages over the regular background noise.
I'd like to know what those distinctive traces are, which is also missing :(
This was probably written by their security team. Security teams are paranoid. They want everything patched everywhere all at once at a severity level zeo. Also, PR. Also, also, if through some lack of imagination, this was somehow involved in an exploit of their services, it would look really really bad. So, CYA.
Cross-tenant would be very surprising! But I don't know enough about their architecture.
It's weird, right? The underlying CNE primitive here, for CopyFail, is not novel. These happen all the time. Why the announcement? Is it just because CopyFail got so much attention?
https://blog.nelhage.com/2010/12/cve-2010-4258-from-dos-to-p...
Also leaving a massive gap like this behind would be a mistake on multiple levels. For example, it might get combined with another exploit that can achieve unprivileged access to some piece of metal, or you can have a disgruntled employee without admin access escalating their permissions on a box they aren't supposed to see all the secrets.
Yeah. TFA mentions datacenters in 330 cities. That's a lot of Linux boxen. And many of those have, by definition, ports opened to the big bad Internet. These Linux servers are running services. They answer to ping, for a start. I even heard some are running DNS servers. Remote local exploits are a thing.
What does CloudFlare prefer: that when the next remote local exploit surface all their fleet is one copy.fail away from privilege escalation to root or that they get the time (seen that they obviously have quite advanced detection measures in place) to detect the intruder before it gains root everywhere?
It's Linux. It's datacenters in 330 cities. Linux powers the world and that's how things works.
I, for one, I'm glad to own CloudFlare stocks since right after the 2022 crash and, for two, I'm happy they don't let their huge fleet of Linux servers with a non-patched exploit.
I'm asking because I don't think they have such an exposure.
Also, Cloudflare hosts some AI services, so it's possible that some consumers are running Python code in their containers, without the wasm sandbox.
You seem so pressed on the fact "why would they even patch this!!!", maybe because its best practice to patch things? You never known what things could be chained together, so you might as well patch this, given its so obviously bad.
I've been a sysadmin/programmer since the mid-90s. Local root exploits are a dime a dozen. If your infrastructure relies upon the tenuous difference between root and non-root accounts, you've already lost. Cloudflare isn't an ISP handing out shell accounts on Unix machines.
So again, yes, of course you should patch your Linux machines. Defense in depth and all that. But the question remains: "why Cloudflare would have been vulnerable to CopyFail?" (in anything but an academic sense). Because I do not believe that they can possibly be relying on the difference between root and non-root account.
It is pretty clear they aren't too concerned about this being a issue for this business, after the first paragraph in bold on the blog:
"There was no impact to the Cloudflare environment, no customer data was at risk, and no services were disrupted at any point. Read on to learn how our preparedness paid off."
As mentioned, you never want to give options to a potential attacker/exploit by keeping known vulnerabilities present in your system. You cannot always predict every single avenue an attack could leverage.
Imagine having a data center with barbed wire fences, guard posts, security and cameras covering every square meter of the facility. You wouldn't just leave a door right open because in theory, people shouldn't be able to walk right in. But why would you willingly leave a door open? Even if the possibility is 0.000001%?
People like you would be the first to turn and say "Cloudflare are morons for not patching this!!! Me and my 1 billion years experience and goat status would of prevented this' when some major Cloudflare hack occurs and it was found that phishing 30 different people and using 9 different exploits (including Copyfail) allowed the attacker to bring down Cloudfare
Ah, then the root/non-root distinction means even less. They don't even distinguish between non-root accounts! Again, I'm not arguing against them defensively patching their systems against known exploits—they'd be crazy not to; just agreeing with Thomas that they can't be relying upon protecting root from non-root accounts as part a normal operational security boundary.
To wit: if an attacker escapes V8, it's unclear that leveraging "Copy Fail" to escape from non-root to root buys the attacker a whole lot more.