Posted by jenders 2 days ago
For swap, the best advice is to disable swap on your desktop, unless <8GB RAM. Really, I've never needed it and you probably won't either.
In the late 90s databases were regularly set up in a way to store their data on raw partitions. There were other types of applications too that required partitions but for databases it was really common.
The practice really only died with OSes allowing apps to bypass the normal filesystem cache.
In Linux this is O_DIRECT and interestingly Linux was super late to the party because Linus hated it with a passion. Now, he was far from the only one, at least Alan Cox and Andrew Morton were skeptical, others as well. Linus was unusually extreme about it and that particular discussion is probably the single source of the majority of quotes people use against him.
Is there really no higher-priority kernel process to prevent total freeze of the system and send a SIGKILL to the culprit process when such a scenario happens?
User-space OOM killers never really worked for me and imo are not a proper solution anyway. This option instead lets you make the kernel OOM killer actually work for desktop use.
Currently have it set to `1000` and it works very well for me (don't remember the last time I had a full system freeze due to OOM).
systemd-oomd should be integrated in systemd, you can configure it to your liking and see if it improves your problem.
Chris Siebenmann discusses when the OOM killer triggers: https://utcc.utoronto.ca/~cks/space/blog/linux/OOMKillerWhen
Chris disables systemd-oomd after it obliterates his X session with no explanation: https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdOomdNo...
> > [...] If the configured limits are exceeded, systemd-oomd will select a cgroup to terminate, and send SIGKILL to all processes in it. [...]
> By having the user@.service template be enrolled in systemd-oomd, Fedora made the cgroup that systemd-oomd would select to be killed be all of your processes (across all of your sessions, if you have more than one). ...
Maybe *Fedora* has fixed or improved in the last 4 years. Or maybe they don't run Fedora.
If you are you using disk swap - not sure why would if you have a SSD, but I once heard some justification for doing that - then install early OOM.
In a nutshell, overcommit. It's more or less broken by design but it's also incredibly practical so pretty much everyone does it.
Couple that with the fact that it's difficult bordering on impossible to correctly determine the culprit. If you've got 16 GB RAM and the user launches 3 processes each of which attempts to use 8 GB who should you kill?
I recommend using the earlyoom if you want more aggresive oom kill:
https://github.com/rfjakob/earlyoom
The README contains a lot of interesting information.
It took a long evening to figure out that it gets earlyoom'd immediately because it tries to allocate too much. Previously the very familiar hitching and freezing was a very easy sign of what kind of issue I was dealing with
The culprit turned out to be the kernel itself, and rolling back to a 6.1 series kernel made the problem go away. I see that Linus's love for vibe-coding is already paying dividends.
If you know ahead of time which programs / processes are at risk of unacceptably high memory usage, check out "ulimit".
And since that happened on all nodes around the same time, it took out the entire cluster. If you are not familiar, random, uncontrolled node restarts in any kind of multi node database or search product are a great way to trigger outages. So, not great.
I've had quite a few encounters with the oomkiller killing processes that were important and didn't need killing. Or as I like to phrase it "killing the one reason this server exists".
These days the way to size a server is "have enough memory to run whatever you need running + at least half a GB for whatever apt might randomly demand at any point". And guess what, memory tends to be expensive in cloud environments so people tend to get vms with as little as half a GB of ram.
Meanwhile zswap (of no relation with zfs) is free performance. Is it a rule that everything starting with a 'z' must be cool?
And if I have a separate otherwise unused ssd then using a swap partition simply is easier than creating a file system for it and mounting it somewhere. Block level backups also are easier.
What would be nice about swap files is if they could dynamically grow and shrink without a special setup.
Way too many variables on Linux for a one size fits all.
One nice thing about swap partitions is that they can be fixed in place even before OS installation.
(Of course, if you’re using swap enough for this to be a huge factor, you probably have bigger problems. But I haven’t done the math.)
Linux can work with TPM but it's never as smooth. (Unless I guess you make it custom for your specific setup.)
echo "/swapfile none swap defaults 0 0" >> /etc/fstab
(ZFS might be described as the love child of lvm and btrfs, 'cept ZFS got there first)
( Very briefly: you hand ZFS a stack of disks (in mirrors or raidz groups, for redundancy) and it makes a zpool. Then you carve the pool into datasets as needed. Each dataset looks like an ordinary directory and takes only the space its files actually use, so you never guess partition sizes again. And each one can have its own compression, snapshot schedule, quota, encryption, and send/receive backups, etc etc etc. )
Performance of zvol is worse than for LVM logical volumes. Last I've looked, zvol for swap is just asking for trouble, up to and including deadlocks. ZFS filesystems do have weird edge cases and incompatibilities, e.g. around free space reporting, quota and the like. And support with software vendors e.g. for enterprise distributions or backup software is spotty at best.
Not true at all.. but most large object storage redundancy has moved to the application layer. There are still workloads that are a great fit but I agree OpenZFS is currently not well suited as a general purpose FS.
But even for the "frequently reinstalled cattle"-usecase, LVM has benefits, e.g. snapshots for backups, debugging and forensics.