Top
Best
New

Posted by Ecno92 11 hours ago

Do not buy RAM, compress(curiousbits.nl)
185 points | 157 comments
varenc 4 hours ago|
As others have said, macOS does this by default and it's quite impressive. I have a long running daemon with a memory leak. By its nature, the leaking memory is very repetitive and easily compressed. Right now this daemon has tried to claim 2.5GB of memory, but it's been compressed to a mere 450MB of actual memory! And because it's a memory leak, it never accesses this old stuff, so it never gets decompressed. Only 20MB of RAM is held that isn't compressed.

Of course, it would be far better to just not have the memory leak...but in the mean time, macOS memory compression is making the leak nearly a none issue.

davidkwast 3 hours ago||
It's like having a room that shrinks everything to 1/5th of its size but you never come back to get anything. It will work until gets full.
cpeterso 2 hours ago|||
A new take on the OOM killer: if a long-running daemon hasn't touched some heap pages for a very long time, perhaps the OS could assume it is leaked and unmap the pages. If the daemon tries to access that memory, let it crash.
kevincox 2 hours ago|||
This sounds like a bad idea. No one has logged in via SSH for a while so the daemon hasn't touched any pages (or only pre-auth relevant pages) then when you log in it gets killed. Or if someone sends a rarely used request to an HTTP server then it crashes accessing some data unique to that request.

Sure, they will get restarted, but not great.

cxr 1 hour ago||
It's a good example of perverse outcomes. Rather than simply following the contract and functioning correctly, we would be:

going through extra effort to accommodate/enable buggy, poorly behaving programs, and meanwhile—

punishing those (and the programmers of those) that are well-behaved and functioning exactly as intended but that now have to contend with evading some heuristic of badness.

dotancohen 1 hour ago||||
Or swap it to a less expensive medium, such as the SSD.
TheSoftwareGuy 1 hour ago||
That’s actually a very old idea, called paging or swapping. And it totally works
01HNNWZ0MV43FF 2 hours ago|||
If the user was feeling stochastic, the OS could theoretically just drop pages that haven't been accessed in, say, 2 days, and send the process a segfault if it tries to access them.

Thus solving memory leaks forever...

bayindirh 2 hours ago||
I believe one can create a kernel module called leaky_ram to make this possible. No?
trollbridge 3 hours ago||
Wait until people find out about swapping.
bayindirh 2 hours ago|||
Oh we found out about it. In the old days, a machine with a load of 57380 (probably not the highest number I have seen) means the machine has run out of memory and started swapping heavily.

Fun days.

Now, we find out when the SSD wears out, in the worst case.

Sharlin 3 hours ago||||
Paging to disk has also become vastly more transparent, and less of a last resort, with M.2 SSDs. No more things slowing down 10x and endless HD noise when out of RAM.
dotancohen 1 hour ago|||
Why the downvotes? I also was thinking that's a perfect candidate for swapping to disk.
hobo123 42 minutes ago||
You can also combine both. Linux has zswap.
Lio 9 hours ago||
RAM compression sounds very useful but the thing I really need extra RAM for is local inference.

Since the models are effectively highly compressed data already my assumption is that RAM compression won't help improve that much (I'm assuming unified memory here as per the Framework Desktop). A bit like trying to compress a JPEG in a zip file.

Could anyone comment on that?

throw-qqqqq 4 hours ago||
The max theoretical ratio of (byte-independent) compression can be approximated using the Shannon entropy.

Most quantized GGUFs have a very high Shannon entropy, so are almost perfectly compressed already. A Q4_K_M quantization is often around +7.90 bits/byte and so is virtually incompressible.

Unquantized models (F16) are often around +7bits/byte for comparison.

https://en.wikipedia.org/wiki/Shannon%27s_source_coding_theo...

hobo123 38 minutes ago|||
Since you need to apply CPU time to compress memory, and also CPU before you can map pages back into process memory (by uncompressing), this is only useful for not frequently used memory, like swap only a bit faster.

I'm not too familiar with LLM memory access patterns, but since the model files are stored on disk, you wouldn't ever keep them in memory anyway (except caching) and just reread them on demand.

RossBencina 7 minutes ago||
> I'm not too familiar with LLM memory access patterns

Dense architectures hit every byte of the model for every token. It's the ultimate memory bandwidth nightmare.

MoE models use subsections for each token and are more amenable to keeping parts in secondary storage, and indeed there are some local llm engines that do this.

vlovich123 4 hours ago|||
You are correct, compression doesn’t help reduce a model. That’s why quantization is inherently a lossy operation although there’s tricks to limit the perceived loss. That’s the only way to further compress the data.

One way to think about it is that if the weights were compressible, it’s a clear signal that better training could make the model smarter as there’s redundant information in the structure that could be used to store other meaning.

The other challenge with compressing weights of course is that you have to decompress them before you do math with them. Dequantizing weights is significantly cheaper and supports random access too (not that that matters too much but it helps in terms of leveraging SIMD techniques to do it in an embarrassingly parallel way)

Powdering7082 3 hours ago||
> redundant information in the structure that could be used to store other meaning.

Consider the induction heads, their W_O are generally expected to be the identity matrix and it is useful to have many of them at many layers (e.g. if you only had one induction head in the whole network you'd only be able to perform one induction circuit per token emitted)

That being said the attention heads are a small portion of the total network size

matja 7 hours ago|||
> A bit like trying to compress a JPEG in a zip file

Interesting comparison, I wonder if there are loss-less compression schemes that are "weight-aware", about the statistical properties of model weights.

Back to that line - because (Huffman coded) JPEG images can be lossless recoded using arithmetic coding, which makes them smaller than if they were compressed with the best general-purpose compression algorithms/tools (e.g. `jpegtran -arithmetic < input.jpg > output.jpg` beats `xz -9 < input.jpg > output.jpg`). AC takes advantage of the properties of the image coding better than general-purpose compression tools on the output of the Huffman coding.

rft 6 hours ago||
The library I use for archiving compressed raw LLM weights ([1] , posted in my other comment) afaik at least reorders the weights to more easily compress them. They also have a paper linked, which I admittedly didn't read, that might have more background and tradeoffs. Likely they will have discussed methods to take advantage of the structure of weights.

[1] https://github.com/zipnn/zipnn#results

rft 9 hours ago|||
I haven't tried it in RAM, but your reasoning matches what I saw when I looked at compressing weights for archiving them. The weights don't compress nicely with standard, especially fast, algorithms. Some more background in this repo [1] including example compression ratios. I would also worry about decompression speed, that gives you a hard bound on the throughput. If the numbers in this repo are accurate, you likely hit the throughput limit of decompression way before you hit the memory limit, even considering the space savings and thus higher effective throughput.

[1] https://github.com/zipnn/zipnn#results

kalcode 9 hours ago|||
I don't know at all. But I've been playing with oMlx and Splash. Using Pi there are compression extensions like the billion-context. I suspect your right.

But all seem to have massive issue boards. What I did see was oMlx release clustering so you can string 3 macs and have 3 times the memory or use one for prefilling accelerating.

Which I thought was neat and what I'll likely poke at instead of getting more ran.

0123456789ABCDE 8 hours ago||
perhaps i'm not pilled enough for this conversation, but aren't you conflating two disparate concepts — model-weights and context-window?
kalcode 4 hours ago||
Yah I think I am conflating it. I was thinking using the ram more effectively to begin to squeeze more, one of those is context size. Less VRAM the less context you can handle, so there always a lot of tuning to find that sweet spot.

So my mind on that.

strobe 4 hours ago||
it interesting because usually on workstation the browser & electron apps it's that holding much of the memory hostage (it's very common to see that browser consuming 20-30gb on 64gb system), so may be if it will be significant compression win on that side it leads to more memory for model but how much performance penalty would be for inference in that case?
pdlclay 3 hours ago||
[flagged]
hedgehog 3 hours ago||
As others have noted zswap is usually a better choice than zram, additionally depending on your situation it may be worth checking which algorithm it's using as there are a few choices (zstd, lz4, etc) and the defaults vary by distro. zstd is the right choice IMO unless you're in a situation where you know better.
jakozaur 9 hours ago||
> Memory compression is not new. If you have a Windows or Mac device you are already using it.

Yes, macOS has had it enabled by default since OS X Mavericks 10.9, released in October 2013. I’m surprised some desktop Linux distributions still require manual configuration to enable it.

For browser tabs, though, I think a better solution is automatic tab closing, using something like the Tab Wrangler extension for Chrome. Collecting tabs comes naturally; we’re much worse at closing them.

hobo123 32 minutes ago||
Brave on Mac (but not Linux IIRC) does this, you can choose how eagerly it will suspend background tabs, although this is somewhat erratic - I've seen it suspend a paused YT video that has been idle for maybe an hour (and not much memory pressure on the system), and then sometimes it takes its time.

In Firefox there's about:unloads where you can suspend tabs in I believe LRU order. I'd really like to see this as a default option (like: only keep N tabs active). Maybe there's an extension for that.

xnx 5 hours ago|||
Chrome has had a built-in tab-sleeping Memory Saver feature for 4 years: https://blog.google/products-and-platforms/products/chrome/n...
benjiro29 4 hours ago|||
"For browser tabs, though, I think a better solution is automatic tab closing" ... the better solution is to actually offload tabs. The issue with Chrome/Edge/... is not your tabs itself but the huge amount of memory loaded for GPU operations.

Do Shift+Esc and watch how large the GPU memory usage is. Often 2/3 to 50% of your actual tab usage.

Ironically, its not the closing of tabs that bothers me, but the fact that tab order changes, what slows my productivity/muscle memory down. And its the same reason for a lot of people why they keep tabs open.

I never understood why we do not have a basic snapshot / sleep that offloads a tabs memory on disk (and dumps the GPU memory), aka a ghost tab. And fast loads when you view the tab. Sure, there may be a small delay but it can save so much memory, while people can keep their tabs open/organized.

Some extensions try to mimic that but its often a crude solution because they do not have deeper access to the webview2 engine.

jve 8 hours ago|||
Wait, Edge on windows automatically puts tabs to sleep when they are unused (https://explore.microsoft.com/en-us/edge/features/sleeping-t...). I think edge weren't even the first one, thus being kinda surprised I see chrome needs extension for that?
evenfrost 8 hours ago|||
For Chrome I recommend The Marvellous Suspender, which can auto-suspend unused tabs. Has been a life-saver for me for quite a while.
jamesgeck0 4 hours ago||
I thought suspend plugins became redundant when Chrome started shipping a built-in auto-suspend. Is there still a reason to use them?
kasabali 3 hours ago||
They provide more granular options on when and what to suspend compared to crude heuristics of the built in feature.
aa-jv 8 hours ago||
>I’m surprised some desktop Linux distributions still require manual configuration to enable it.

Its a user choice and should be a user-facing knob because there are times it is inappropriate for use, such as in safety-critical circumstances, certain realtime realities, and so on. It should probably be on by default for desktop Linux, but off by default for every other application a distribution may be targeting ...

flowerlad 2 hours ago||
In the 1990s there was a third-party add-on for Windows called SoftRAM95 [1] that doubled your RAM using compression. That one turned out to be a scam though [2].

[1] https://en.wikipedia.org/wiki/SoftRAM

[2] https://devblogs.microsoft.com/oldnewthing/20211111-00/?p=10...

jmakov 1 hour ago||
Looking at the solution AMD bought, looks like Linux has a lot of room to improve regarding moving (compressed) inactive pages to disk: https://www.amd.com/en/products/software/mext.html.

If my understanding is correct, both ZRAM and ZSWAP write uncompressed pages to disk. And both don't pack the pages together meaning even if we'd write compressed pages (let's say they compress 5:1), we'd still write the original size to disk. This is crazy.

zamalek 43 minutes ago||
> If my understanding is correct, both ZRAM and ZSWAP write uncompressed pages to disk.

ZRAM added compressed_writeback in 7.0, it needs to be explicitly enabled.

jmakov 37 minutes ago||
Yes, but if a 4k page compresses to 1k, a 4k block on the disk still gets used: 1k + 3k of zero padding. Meaning if 100GB of data compresses 5:1, 100GB needs to be written to the disk instead of 20GB.
quotemstr 48 minutes ago||
FWIW, DAMON (https://docs.kernel.org/mm/damon/design.html#operation-actio...) has supported explicit control over Linux working sets for a while now. For whatever reason, ISTM few people know about it, but it's a nice subsystem with a lot of policy flexibility.
zamalek 44 minutes ago||
> Compressing memory does come with small, but not really noticeable, downsides. It takes a little bit more from your CPU and adds latency to some processes.

I recall reading an article around the time the Windows team added it (back when I still used the OS). Their compression algorithm was typically faster than the latency to the prevailing disk type at the time (SATA SSD, most likely) - at the time it had zero downsides. I'm not sure if that still holds true.

On Linux it generally doesn't: off-the-shelf compression is used, which do not outpace storage access latency.

noodlesUK 48 minutes ago||
I think it's a great shame that kubernetes really doesn't like systems with any kind of swap on (including zswap). If memory compression were more commonly used I think many workloads could survive on smaller systems.
2001zhaozhao 25 minutes ago||
I do this and also buy old used 16GB Optane sticks to use as swap. These are $7 a pop on ebay. They're way lower latency than normal swap and you can swap data as much as you like without worrying about wear.
cogman10 5 hours ago|
Don't use Zram, use Zswap. Especially if you want to swap to disk.

https://chrisdown.name/2026/03/24/zswap-vs-zram-when-to-use-...

mrob 3 hours ago||
I never want to swap to disk. If I'm ever in that situation, it means I did something wrong, so I want my userspace OOM killer to trigger and inform me of my mistake immediately instead of wasting time and SSD endurance delaying the inevitable. But zram with lz4 compression is effectively free RAM in most circumstances, so there's little reason not to enable it.
quotemstr 49 minutes ago|||
You absolutely do want to swap to disk. There are lots of non-file-backed cold pages that are pointless to keep in RAM forever. You could argue that these pages shouldn't exist, but they do, so why not shove them to a disk?
mrob 25 minutes ago||
I want instant access to my rarely used pages just as much as my frequently used pages. It's only pointless to keep in RAM if it's literally never used. If we knew how to identify those pages we could just avoid allocating them in the first place.
quotemstr 3 minutes ago||
Your system is still swapping all your file-backed pages (e.g. .so files) unless you're mlockall()-ing everything. If you pin non-file-backed pages in RAM by running without a swap file, the system has a harder time keeping your frequently-accessed file-backed pages in RAM and can't do as good a job as optimizing your usage patterns.

All you're doing by running without swap is pessimizing yourself.

dsp_person 3 hours ago||||
Good point, dumping to SSD isn't ideal. I think I'm gonna start trying to set memory limits on programs with systemd-run so they can self-limit themselves.
01HNNWZ0MV43FF 2 hours ago|||
For today's 10,000, do `sudo apt-get install earlyoom` and thank me later
teravor 2 hours ago|||
> Especially if you want to swap to disk.

if you don't want to swap to disk you should not use zswap.

zram is ideal in that scenario, though it may complicate OOM killer considerations (you may want a userspace one).

ciupicri 5 hours ago||
I've tried for running bsdiff and zswap was slower and spilled out data to the disk (which would wear an SSD).
More comments...