Top
Best
New

Posted by farlight 3 hours ago

Btrfs/ZFS/bcachefs under workloads classic benchmarks skip(bartosz.fenski.pl)
70 points | 67 comments
fenio 1 hour ago|
The author of the benchmark here. I went over some comments and I'll try to tackle them here. I'm pretty clear that GH runner based benchmark is far from perfect due to noisy neighbours etc. Thus every test first is running so called calibration... to reject completely unreliable VMs. I'm fully aware that this can't completely fix the issue. Can limit it but not fix. But as of now there are 593 runs recorded so average should still be quite meaningful.

Having that said I'm desperately trying to get REAL hardware to run that benchmark. With some successes ;)

Few months ago I got Hetzner machine from Kent Overstreet and I was able to finish 3 runs before machine died... Results: https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/

Currently I've got even more interesting machine with tons of disks and I'm running new set of benchmarks but it's really in its initial stage.

https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/ 2nd run in progress... one run on REAL hardware takes much more time than on GH runner so it's slow.

But this new hardware has also so many disks that the plan is to try also more complex, tiered cache topologies. I'm working on it.

I'm happy to answer any other questions, sources of every piece of this benchmark are freely available and I'm not saying they are 100% correct. I'm open to improvements.

koverstreet 1 hour ago||
I went back and forth with Hetzner a couple times, I think we just got a bad machine :)

I've been saying it for months, but eventually I'm going to move the automated builds off the 48 core monster and we'll be able to use that for automated perf testing too. The machine we just got has spindles for EC perf testing, but the Hetzner monster has very high end enterprise ssdd.

Also, just got done with the Rust for Linux conference, still not home but here's slides that still need reformatting: https://evilpiepirate.org/~kent/Kangrejos-2026-bcachefs.pdf

Joel_Mckay 1 hour ago||
Should include:

1. Dual Ext4 + external 32GB journal X4 pcie SSD (the prior winner of benchmark surveys)

2. Bare F2FS after a trim and SSD vendor software cache flush operation (it should be slower, but knowing how much slower on identical hardware could be interesting.)

3. DRBD across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary)

4. CephFS across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary)

Best regards =3

irusensei 1 hour ago||
BCacheFS is the best Linux filesystem now that storage costs a premium.

You can mix devices of different sizes and types on bcachefs. You can have foreground and background devices to balance performance and also different compression settings for foreground and background transactions.

You can set replicas=N to the individual file or directory on bcachefs. For example files you can just re-download or re-build. Likewise you can set a higher number of copies to important files.

suncore 51 minutes ago|
Sure, when it's mature...
mkeedlinger 34 minutes ago||
Yeah, and I think Overstreet might be going hard on AI, so not sure how that might effect the project
koverstreet 24 minutes ago||
If you have "refine until it's perfect and don't screw with things you don't understand" thoroughly ingrained, along with the dangers of overconfidence, you'll do fine with AI.

Not everyone gets it though, that's for sure.

And, if you want to know if it's mature, I'd trust the user reports over the one liners :)

bhaney 38 minutes ago||
Seeing great results from bcachefs just makes me more sad that Kent and the other kernel devs couldn't come to an understanding to keep bcachefs in-tree. I want to use it for my storage arrays so badly, but I'm still stuck with btrfs as the only available in-tree filesystem with modern features.
koverstreet 26 minutes ago|
NixOS. You can't go wrong.
Farmadupe 2 hours ago||
> CI runs use loop devices on shared ephemeral VMs (one VM per filesystem): compare shapes and ratios, not absolute MB/s. Each job records a host-calibration anchor — see the table.

I think if you're not using baremetal for such tests, it's likely that the results are simply not comparable at all? What if another tenant is also using the disk?

walrus01 2 hours ago|
It's a fair point but it's also possible the person running the tests has a dedicated test hypervisor for this , so that different configurations of filesystems and VMs can be created and destroyed quickly in an automated manner.

If it's something as simple as a KVM hypervisor that only runs 1 test VM at a time (with no other load from anything else other than the basic systemd daemons, ssh daemon etc running on the hypervisor), the results could be very close to bare metal.

I can see it being very time consuming and annoying to do repeated manual bare metal OS installs and new partitioning/filesystem creation for such a large variety of tests.

The author does also say that performance isn't really the main thing but rather, data integrity:

https://github.com/fenio/modern-fs-benchmark

toast0 2 hours ago|||
> I can see it being very time consuming and annoying to do repeated manual bare metal OS installs.

Well don't do that then. There's lots of other options. Probably the simplest is a single bare metal install on a simple filesystem on one device. run the filesystems under test on other storage dedicated to testing.

You could also boot into a network install and use local storage exclusively for testing.

Farmadupe 2 hours ago||
Yes exactly. The issueThe epherrality of the VMs isn't an issue, it's the _shared_ part that's the concern here. Going by the fact that the kernel is listed as "kernel 7.0.0-1012-azure" I feel like it's a fair risk that there may have been noisy neighbours.
fenio 1 hour ago||
Obviously they could and probably were noisy neighbours. I'm not trying to hide that fact. Initial step for every benchmark is test of underlying device to at least reject completely unlucky cases. Also after almost 600 runs average is probably more or less correct...

Also take a look at tests on real hardware. There are not many of them but there are some. I pointed to them in my first answer.

fenio 1 hour ago||||
For the main linked benchmark there are no dedicated test hypervisors. It's all based on GH runners with all the limitations and quirks that came with it.

Real hardware is used in: https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/ https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/

But unfortunatelly it's much more limited number of actual runs. sas-hdd is still in progress so numbers for it should increase over time.

Farmadupe 2 hours ago|||
> compare shapes and ratios, not absolute MB/s

In this case, given that the author's own disclaimer (above) already disclaims the numeric readings, I'm not sure how it's possible to make any inference on "shapes and ratios" derived from the numeric readings.

viraptor 11 minutes ago||
It could use some explanation for the exact setups. Like how many drives are there in each raid, what does the ec config actually look like, etc. As far as I can see it's not described.
loeg 2 hours ago||
What is md-raid10 doing that is so much worse than lvm-raid10? In terms of "I/O" and "responsiveness." It's not really obvious to me from either the linked page or https://github.com/fenio/modern-fs-benchmark . In principle they should be similar?
jmalicki 1 hour ago||
[flagged]
nullc 5 minutes ago||
It'll be interesting when people start applying similar effort that is going into LLM search for security vulnerabilities to finding conventional reliability/performance bugs.
sippingabonedry 1 hour ago||
So two filesystems that are essentially shunned from the Linux kernel and permanent second-class citizens, and one that was removed from Red Hat and has a questionable history of reliability. Oh boy which do I choose?

I'm saying ZFS on another OS.

petre 1 hour ago|
xfs/lvm-raid10 or ext4/lvm-raid10, obviously.
magicalhippo 1 hour ago||
> Every push/2-hourly cron builds each filesystem across 4 loop devices backed by sparse files, runs the suite, and publishes a results table in the job summary plus JSON artifacts.

I get that real hardware costs (author mentions EUR 70 a month for a suitable server), but without at least a baseline snapshot comparison run between real hardware, both SSD and HDD, and the sparse file-backed loop devices, it's hard to take much away from this.

Sadly the AI apocalypse isn't making stuff like this easy to do as a hobby.

skerit 2 hours ago|
Oh, so bcachefs is doing pretty well.
tarruda 2 hours ago|
Except for the fact that the developer has sabotaged the project into being removed from mainline?
irusensei 1 hour ago|||
That might be the best thing happened to the project since now development can happen at its own pace without the clicky bait influencers.

In fact they delivered the erasure coding for parity raid back in march this year.

The thing is that as soon as you seriously give a chance to Bcachefs you see how good it is. I can only tell you that mixing different device tiers and having a per-file/directory replication setting is a god send specially in these times where storage costs more than gold.

tarruda 1 hour ago|||
> The thing is that as soon as you seriously give a chance to Bcachefs you see how good it is.

I'm pretty sure Bcachefs is amazing and better than Btrfs. I also think Zfs is amazing and better than Btrfs. Even so, I still use Btrfs because I know it is guaranteed to always be present on any Linux without any effort on my part.

> That might be the best thing happened to the project since now development can happen at its own pace without the clicky bait influencers.

A better approach might have been to just paused mainline merging instead of forcing being kicked out?

Eg "Hey Linus, Bcachefs is still in early development and I need to merge changes in a pace that is not compatible with Linux development process. So I'm going to pause for a while now and once it reaches maintenance status I will focus on submitting patches in a healthy pace that you can digest".

ssl-3 8 minutes ago|||
> I still use Btrfs because I know it is guaranteed to always be present on any Linux without any effort on my part.

I used to think that about ReiserFS, too. It was in the mainline kernel, development was snappy, and it solved some performance problems. I used it all over the place.

Things then subsequently... changed. :-/

nvme0n1p1 58 minutes ago||||
> Even so, I still use Btrfs because I know it is guaranteed to always be present on any Linux without any effort on my part.

100%. My system is rock solid and the last thing I need is rolling the dice after every update on whether my system will boot. https://www.reddit.com/r/archlinux/comments/eywcp7/linux_551...

I'm impressed with bcachefs's accomplishments though, and if they ever reconcile with the kernel I'll surely give it a fair shake.

irusensei 1 hour ago||||
I think in a perfect world they should had put someone in between to mediate and curate patches while providing DKMS for urgent patches.

As for BTRFS I think its also pretty good. Its just that I have the impression its development is guided by the needs of its sponsors and sadly for us META doesn't need RAID5.

koverstreet 1 hour ago||
Meta doesn't have anyone working on btrfs anymore, it appears to be two guys at SuSE and drive bys.
koverstreet 1 hour ago|||
I was very up front about where we were at.

A lot of things were tried, people did try to mediate.

The particularly galling thing though was when I finally started looking - post split - comparing bcachefs PRs to other subsystems and especially XFS - I was being more conservative with what I considered a critical bugfix.

There was never a clear statement on what the issue was. What you guys got in public was about as much as I got.

All I can say is - going fast when you're stabilizing and getting bugfixes out the door is what you can and should be doing when you've invested in test coverage, test automation, keeping the codebase clean and asserted, and building up a community that works well together on testing and shaking things out.

I genuinely do not know what they were thinking.

novafunc 1 hour ago|||
bcachefs was already working at its own pace prior to being accepted in the kernel. It could have continued doing so for years until it was really "ready".

Instead it got kicked out because Kent constantly ignored the kernel's contribution rules and is unlikely it will ever be accepted back into the kernel.

irusensei 55 minutes ago|||
That would be a shame since there is nothing else over there with the same set of features. Disregarding drama, I'm telling you it’s that good.
koverstreet 41 minutes ago|||
I'd really appreciate it if we could drop the FUD over contribution rules. There are no such rules, it is explicitly Linus's way or the highway, and I already replied to that elsewhere.

And it went in when it did because Redhat was pushing for it and claiming to be supportive - but that never materialized. They wanted to get something for free without investing, or putting in the absolute bare minimum.

A _lot_ of people were saying publicly and privately "dear god yes we need something better than btrfs" - but no one from the existing kernel community was interested in stepping up.

Community's still growing, though. A lot of people have gotten active in making sure bcachefs actually works well for people end to end, and there's a hell of a lot more to shipping a filesystem than just writing kernel code.

pantalaimon 18 minutes ago||
You can't let Reddit guide your technical decisions.

The FS was marked experimental, so there is no urgency in fixing bugs or providing features in a certain cycle. Everyone using it knows what they got themselves into. You can still provide the DKMS module for faster fixes and features for anyone who wants to use BCacheFS more seriously for the time that the upstreaming process takes, but eventually it would have all been on mainline.

Asahi is taking a similar approach where they have their downstream kernel and push things upstream once they are mature.

That means the upstream kernel is not useful for running on that hardware now, but things are moving there eventually.

tombert 1 hour ago||||
It's relatively easy to get it working as a kernel module at least. I got it set up on a NixOS box without too much trouble.
petre 1 hour ago||
Not something I'd want to do with my filesystem though. ZFS is different story it it isn't on the root fs because it might be packaged by the distribution.
tombert 57 minutes ago||
Fair, I still don’t use it on my main machine, just a shitty laptop that I keep around to play with.
AceJohnny2 2 hours ago|||
that's not necessarily a sabotage.
eikenberry 1 hour ago||
Sabotage might not be the best word, but it hurt trust and adoption.
koverstreet 1 hour ago||
It's just been a lot less drama within the project since the split.

I do have a lot more pull requests to merge than I did before. I don't know if you want to count "Kent isn't reviewing PRs fast enough" as drama :)

More comments...