Posted by jwilk 5 days ago
I have many opinions regarding this situation, but it mostly doesn't matter. AUR staff and AUR helper developers will figure out what they want to do, hopefully they will find a good approach.
But what I personally take away from this is simply that it has become worth it to target desktop Linux with malware. Or at least, moreso than previously. It is perhaps a good sign in some ways that the desktop is starting to be taken more seriously.
The bad news, of course, is that the Linux desktop is a bit of a train wreck in terms of security hygeine. It's getting better, and Linux does have the advantage of having some powerful primitives to exploit, but the desktop suites come from a totally different world, and I fully expect we'll also see more malware propagated through KDE's New Stuff integration (which goes through Pling.)
In general things that are not part of your distro's supported repos (KDE's AUR, language package installers like npm and pypi, Ubuntu PPAs, etc.) seem to present far more of a risk.
Absolutely the wrong conclusion:
1. This is a super low hanging fruit attack
2. The ROI is significantly higher than the cost of attack
3. The cost of the attack is now drastically reduced due to LLMs (not that they necessarily mattered here but hard to rule out).
It’s less about the Linux desktop where Ubuntu is dominant and more that Arch security is so bad regarding how they maintain the AUR. Seriously it’s worse than Swiss cheese in terms of putting up roadblocks.
I still don't understand what people expect Arch to do here? It's a user-contributed repository open for anyone, Arch maintains their own official repositories that are separate from AUR, what would need to change in the maintenance of the AUR for you to consider it to be "properly run" or whatever, and it doesn't stop serving its core purpose anymore: to allow any user to upload any PKGBUILD?
If you're advocating for not allowing users to upload their own PKGBUILD anymore, then it stops being the AUR, and for the people who agree with that, they can already exclusively use the official repositories instead of touching the AUR.
In true Arch fashion, the security is up to you here, review stuff before installing 100% unknown and random 3rd party software from the internet, as always. If you don't want to review anything? Stick with official stuff, and you won't have issues.
So, firstly it is their responsibility. The consequences are a direct result of their policies.
Example of changes:
* orphaned packages don’t need to be adoptable.
* doesn’t have to be a flat global namespace
* they could have offered the cooldown capability a long time. This isn’t an area they focus on until they’re absolutely forced into it through sheer embarrassment of the scale of the attack.
* they could be running scanning tools on every change to try to catch low effort attacks like this
But sure, if you throw up your hands and say “we’ve tried nothing and nothing works” then you shirk all responsibility. And maybe shutting down the AUR is the right move if the current incarnation is so bad and the people running it don’t know how to secure it.
Think of it this way - the NPM and cargo registries handle way more traffic and visibility and doesn’t have any issues like this. There they have to deal with more complicated supply chain attacks. Ubuntu and Fedora take a completely different tack with user-contributed packages that are namespaced and better sealed.
And part of this of course is that yay and ilk don’t do proper sandboxing. But that’s just more shirking of responsibility by the AUR maintainers to force the community to try to do this and failing to treat this as an end to end problem they need to own
Those things sound worse for us who actually use the AUR the way it's meant to be used, being able to "orphan" packages for new maintainers to pick up bring us long-term stability. And since we review random 3rd party software we install from the internet, who does the actual edits doesn't really matter, as long as it's the right, simple little changes that updates usually are.
It's easy to complain about other volunteers to non-profit projects are not doing enough, but truth is that there is always a lot of stuff to do as an volunteer, and while you try to fight fire X, people scream about fire Y, and vice-versa, depending on which one made the news most recently. Sure they should be scanning things, sure the official repository should be super fast and always available, sure every package should always be problem free, but it's a human project driven by humans essentially for the love of the project itself, in one way or another.
Cargo in general have the benefit of being new (shoulders on giants and so on), and being made by people who knew what they were doing. NPM has had a long struggle with a lot of stuff though, and is today maintained by one of the largest companies in the world. I'm not sure they're really comparable here.
I do agree the process for which orphaned packages get new maintainers needs to change, obviously shouldn't be possible launch such automated attack. I don't agree with that the feature as a whole should go away, I do have my own packages that depend on other AUR packages, surely many of them have through the years changed maintainer, but shouldn't mean I need to suddenly start using a different package, as long as I continue reviewing the updates.
This is the problem - the AUR has outgrown this mindset and this resistance to recognize this fact is precisely why this problem will keep coming up. You’re essentially there in some ways without admitting it because of new account creation is disabled. The next attack vector will be taking over existing accounts that aren’t used.
Think of it this way - Arch is a niche distro within a niche desktop OS and still it was cheap enough of an attack that it was worth it. It’s a cultural problem and your mindset is precisely why this will keep happening. As an Arch user I’m honestly embarrassed and I’m going to be looking at distros that aren’t user hostile like this.
It has outgrown the mindset that AUR is for people who use and follow the advice of Arch Linux? Or what do you mean? What I'm describing is a workflow you can apply today, apply it equally to all packages, and it stops 99% of the hacking attempts and the remaining 1% wouldn't matter if it's via AUR, NPM or Cargo, same issues remain.
> It’s a cultural problem and your mindset is precisely why this will keep happening
I agree that "Anyone can automatically take over 100s/1000s of packages as a maintainer" is a problem, I don't agree that it's a deeper problem than that. Limit each user to be able to take over one package per month, and suddenly we get all the same benefits we have already, + we fix the current issue.
No need to trash the entire AUR when there is one specific feature broken, just fix that feature, then continue your pragmatic life as before.
> As an Arch user I’m honestly embarrassed and I’m going to be looking at distros that aren’t user hostile like this.
To be fair, if I ended up misunderstand something so deeply that I didn't realize how to actually use it, I'd be embarrassed myself as well, strong of you to at least state so publicly. I'm happy you at least figured out that Arch Linux isn't for you, and you start trying to find a distribution that fits you better, rather than going through a tough period of time trying to change something into a direction it isn't even aiming for.
The AUR is just as safe as installing through a random shell script from github - that is to say: not safe at all.
I don't have all the answers. I hope people will consider Nixpkgs as a powerful model: even though it has many thousands of PRs unmerged at any given time and requires substantial effort, it still operates at an absurd scale and yet has a higher bar to infiltration with malware, both due to build sandboxing and due to the review process. Whereas AUR maintainership is mostly handled individually per-package, anyone can submit changes to a Nixpkgs and try to get it reviewed, even if the maintainer of an individual package is long gone (though we usually give maintainers a week or so to take a look at a PR even once it has been reviewed.) This entirely gets rid of both the need and incentive to have a process to replace an orphaned package's maintainer: writing yourself in as the maintainer mostly just signs you up to get pinged for issues and PRs, it doesn't grant you special permissions to bypass the normal review process.
A giant monorepo of PKGBUILDs with a similar maintainership model plus makepkg sandboxing would be fantastic. makepkg sandboxing would be nice just for ensuring that packages correctly declare their dependencies.
I think the sandboxing thing has a better shot at actually happening, since redesigning the entire AUR around a completely different model is probably an unreasonable leap. That'd still be a pretty good improvement, though.
> It's getting better, and Linux does have the advantage of having some powerful primitives to exploit, but the desktop suites come from a totally different world,
That said: the attacks on Snap and Flatpak are bound to be less interesting; it doesn't share the same mechanics that make AUR scary. I still hope that Linux infrastructure vendors work to come up with good ways to "raise the floor" in these cases.
A bunch of common yay commands also return back the last updated time of a package thanks to https://github.com/Jguer/yay/pull/2846.
Nothing protect you from a `yay -S foo` install and its dependencies. So this is not a guarantee or enforcement of a minimum release-age.
Actually writing this reply I went ahead and pointed that out in an issue at: https://github.com/Jguer/yay/issues/2883
This is a good thing, because the warning about checking everything you download from the AUR, which has always existed, is now actually "enforced". People respond to consequences.
But unless I've badly misunderstood something, the key thing that made this attack possible is this "orphaned" thing that lets you grant write access to an existing package to the first person who claims it, without any control over who that is. I don't see how this could ever be a safe thing to do, I'm not aware of any other package repository that has it, and I struggle to guess what whoever built it was thinking. If AUR just turned off that misfeature, they wouldn't be having this problem.
(The article quotes someone involved as saying that the "orphaned" feature is good because good actors can also use it, but that seems irrelevant if it also opens up an unmitigable machine-takeover vulnerability. World-writable single-namespace systems like Wikipedia work by having humans proactively checking for bad changes, and also by it not being that bad if a page is briefly defaced, since you can't push malware to users' machines that way.)
I know that for AUR there was a specific list of affected packages (that I checked, and haven't installed any of them), but I'm interested more in a general way. It could be from AUR, npm, or many other sources. Some malware could break and lock immediately the system, but other could stay there silent for months, so how to find out if there is any?
I haven't run an antivirus since I last used Windows 20 years ago.
Ultimately I'd say it depends on your risk profile, but using something to actively approve/deny network connections on your local machine, is a great start that'd defeat most of these simpler "exfiltrate information ASAP" malware that seems popular at the moment.
If you have a list of good CLI utilities, you could run them in a bash script (e.g., network-monitor.sh), which would run in the background, and then redirect the output data to another file (e.g., network-monitor.txt). The key concept here is "baseline" -- you need to know what normal baseline network activity looks like, so that you can identify anomalous behavior. The way to establish a baseline is to gather a lot of data from the system.
The following are a few useful command line utilities to use for a host intrusion detection system (HIDS) using a simple network monitoring bash script. However, I am not sure exactly how to tweak the options. Also need to find a way to check for data exfiltration:
-list open ports and processes that own them: netstat -lnp
-show open network ports: lsof -i; netstat -an | grep -i listen; netstat -nap
Then it would be relatively easy to write a python script with regex tools to parse the network-monitor.txt file, establish a baseline, analyze the data for patterns and search for anomalous behavior.
Besides network monitoring, there are other command line utilities you can use to check the system for possible intrusion, which you could run in a separate bash script as part of your Host Intrusion Detection System (e.g., hids-users.sh):
-show members of root group: cat /etc/group | grep root
-show users logged in: w #if you are the only user, you should not see more than one account logged in
-search for all accounts with UID of 0: grep :0: /etc/passwd #ideally there should be only one UID of 0 on the system, but attacker can create more.
-check that daemons who never log in have * or !, meaning no passwd: cat /etc/shadow
-look for orphaned files, possible sign of attacker temp account deleted: find / -nouser -print
-search for new user accounts that are not part of regular build: sort -nk3 -t: /etc/passwd #sort numerically third column (UID), colon delim (-t:)
EDITS: several small changes for clarity and also in response to comment below
Don't do that, you can't rely on "watch for anomalies" with your human eyes.
Either you setup something that notifies you after the fact, or you outright block all incoming/outgoing connections until you approve them. Mentioned elsewhere I think in the thread, I think both OpenSnitch, Little Snitch and PiHole can help you with all of these things.
But don't assume you can "watch for anomalies", automation and/or gated access is probably the way to go.
https://www.reddit.com/r/linux_gaming/comments/1u34pe3/comme...
Yes, I agree, that's a good call. I would not try to check for anomalies manually with meatware. I would parse the data with python regex tools to establish a baseline and search for anomalous patterns.
I edited my post to reflect the change you suggested.
In this particular malware campaign, the malware contained a rootkit which hid precisely some of its activity:
https://github.com/gustavo-iniguez-goya/decloaker/discussion...
That'd be the role of an IDS (Intrusion Detection System). Things like file signatures of your entire system being saved to another machine (for example an offline/airgapped one) beforehand (for example by plugging your main machine's SSD as a secondary drive on the airgapped one).
If you suspect shenanigans, you take your machine offline, you remove its SSD (your BIOS/UEFI is also a concern), plug it to your airgapped machine with the IDS: it compares all the files (binaries, config files, etc.)' checksums with the past ones.
It's a bit of a lost art but it could make a comeback seen what we're facing, now nearly on a weekly basis.
Some distros have a way to check for file integrity as part of the package manager: but you can't trust the infos coming from the machine itself if it's been compromised.
Mitigation Tool: https://github.com/cookiengineer/antimiasma
Blog Post with details: https://cookie.engineer/weblog/articles/malware-insights-mia...
Some people (many of which don't even use Arch itself) have been treating and advertising it as something different. It's not a software distribution method meant for normal computer users.
I do not think this something you can escape by switching distro.
Ubuntu actually has first-party repositories with proprietary codecs.
Nixpkgs is a pretty comprehensive monorepo of packages with a more normal review process than the AUR, and it includes non-free software as well, plus the model with flakes for third-party stuff is that you trust individual publishers for their little repos rather than one giant grab bag repo of unreviewed content like the AUR.
RPMFusion for Fedora kinda has a similar profile, in that it's a shared repo for various things unsuitable for the main one, but it follows more or less normal Fedora packaging and review standards, doesn't it?
Supply chain attacks are possible everywhere and some distros have particular weaknesses, but the AUR really is pretty much uniquely bad here.
That being said, still one namespace. Once you add an overlay it can replace any package it wants.
It's also Gentoo so too hard for most people to figure out.
Namespacing is the solution, and as mentioned in the article some ditros do indeed have namespaced user repos, like Fedora's Copr. The trust model of a flat namespace user repo is completely broken when the maintaining user can change at any moment.
OBS is more like Ubuntu's Launchpad or Fedora's CO0R than the AUR. Random strangers can't take over the packages of others just because they go idle, and it's a bunch of separate repos, not one. Totally different trust model.