Top
Best
New

Posted by stereo-highway 8 hours ago

Diskless Linux boot using ZFS, iSCSI and PXE(aniket.foo)
106 points | 54 comments
deathanatos 5 hours ago|
> UEFI fixes that to some extent, but it’s a pain to maintain the UEFI entries manually and change them every time the kernel updates.

… you don't have to update the UEFI entries every time the kernel updates. (I guess you might if you do like a kernel w/ CONFIG_EFI_STUB, and you place the new kernel under a different filename than what the UEFI boot entry point to then you might … but I was under the impression that that'd be kind of an unusual setup, and I thought most of us booting w/ EFI were doing so with Grub.)

yjftsjthsd-h 5 hours ago||
Even if you do CONFIG_EFI_STUB, there should be a post-update hook to automatically call efibootmgr.
nubinetwork 1 hour ago|||
I have 2 entries, /efi/current.efi and /efi/old.efi... when I upgrade, I copy current to old, and copy my new kernel out of /boot as current and reboot.
nicman23 4 hours ago||
or just copy the latest kernel to something like /vmlinux and /initramfs
creshal 2 hours ago||
Or use UKI and throw the current kernel to /efi/boot/bootx64.efi; there's plenty of solutions to sane bootloader/kernel management if you're willing to invest 15 minutes into the topic and not act like it's scary and complicated (it really is the opposite).
cwillu 2 hours ago|||
Grub2 is scary and complicated. Remove grub from the equation, and all the scary goes away.
nicman23 22 minutes ago||
grub is just a operating system. it is quite good when shit hits the fan
nicman23 21 minutes ago|||
i never got it to work
cwillu 2 hours ago||
Friends don't let friends use grub.

rEFInd is _so_ much simpler: one efi entry, one text config file in the efi partition, nothing that needs to change when the kernel updates, and no massive pile of templating and moving parts to mysteriously break dumping you at an impenetrable grub “rescue” shell.

mdhen 41 minutes ago||
Systemd-boot is my choice for any simple boot scenario. Love it. Agreed that grub is a mess
DaSHacka 1 hour ago||
There's also systemd-boot, which seems to be getting more popular.

https://systemd.io/BOOT/

yjftsjthsd-h 5 hours ago||
Nice. I'm extra fond of ZFS backed network root filesystem, because it lets you put an OS on ZFS without needing to deal with ZFS support in that OS. (One of these days I want to try OpenBSD with its root on NFS on ZFS, either from Linux or FreeBSD.)

Does anyone have an opinion on iSCSI vs NBD?

guenthert 3 hours ago||
Well, iSCSI is a standard, so chances are better that it's supported in a non-Linux OS, e.g. MS Windows. Years ago I booted a Windows (7, iirc) client that way, but gave up on it (too much hassle and performance limited by the network) when SSDs became cheap.
Modified3019 4 hours ago|||
I don’t have direct experience, but when I looked into it my takeaway that NBD was unable to reliably deal with network interruptions as well as iscsi.

https://forums.gentoo.org/viewtopic.php?p=4895771&sid=f9b7ac...

https://github.com/NetworkBlockDevice/nbd/issues/93

Whether that’s the case with the latest version, I don’t know, but it’s something you might test if you choose to try it.

nubinetwork 1 hour ago|||
I think you need NBD if you're going to use glustre, but I could also be thinking of ceph.
jaypatelani 3 hours ago||
You might like https://smolbsd.org/
yjftsjthsd-h 3 hours ago||
Well yes, I do like that:), but I don't see the connection to this thread?
MohammadAbuG 8 minutes ago||
This man is diskless
Tepix 4 hours ago||
This could be an interesting setup for booting off a NAS like Synology or QNAP. I haven't really used iSCSI, it's intimidating how much prep this takes...
rwmj 3 hours ago||
iSCSI seems intentionally obscure. One of the improvements I made to NBD was invent a simple, standardized URI format so that you can specify servers easily, eg:

  nbdinfo nbd://server
  nbdcopy nbd://server:2001/ nbd+unix:///?socket=/tmp/localsock
https://github.com/NetworkBlockDevice/nbd/blob/master/doc/ur...
burner420042 3 hours ago||
The 'target' moves slow so once you learn it, it all stays relevant forever.

... And it's very, very fun.

Tepix 2 hours ago||
Does it offer performance advantages over NFS root?
a96 2 hours ago||
I kind of expect the performance is worse, but one neat thing is that iscsi is a block device, so you could run e.g. disk crypto, volume management or whatever on it. Not to mention any FS. And you don't need to deal with NFS or RPC.
anonymousiam 5 hours ago||
I've done a lot of headless/diskless stuff. I haven't done much for years, because my NAS only has gigabit Ethernet ports. I can cascade them and get four Gbps downstream, but it's still painful.

I have recently upgraded my house to 10Gbps Ethernet, with only one room still stuck at gigabit, and unfortunately, it's my main office. I'm working on getting the drop there now (literally, just taking a break here).

Even once I'm done, accessing an iSCSI drive over 10GbE will be 4-8 times slower than a local NVMe drive, but it will sure be a lot better than it was!

Ideally, I could run VMs on the NAS and have great performance, but that's another hardware upgrade...

olavgg 2 hours ago||
Using a proper NIC (Chelsio) with their iSCSI accelerator will boost your iSCSI performance significantly. Another alternative is Mellanox with RDMA. You need CX4+ for optimal performance over TCP/IP, while the cheap CX3 is excellent with IPoIB. If you have a lot of packet drops and retransmissions, another option for boosting iSCSI performance is getting a network switch with a lot of memory for packet buffering. This helps with incast congestion. There are special switches with gigabytes of memory built for this.

NVMe-oF is the best protocol with least overhead for network drives, with a proper setup you lose only 10-20% latency compared to local disk even with Intel Optane. Throughput should be almost similar.

ReDress 4 hours ago||
Really I wonder how this turns out to be diskless while you're clearly accessing a disk/drive over the network. Shouldn't we refer to this as network boot?
pdpi 3 hours ago||
It's diskless from the point of view of the device being booted.
dhash 5 hours ago||
something worth mentioning here is that iSCSI is quite unhappy on congested networks or packet loss caused by incast traffic.

to make this actually work well, consider modifying your switches QoS settings to carve out a priority VLAN for iSCSI traffic

fragmede 5 hours ago|
or a north-south/east-west architecture, so there's an entirely separate network just for iSCSI. Control plane vs data plane.
tehlike 5 hours ago||
I used similar ipxe setup for robotic cluster - every robot booted from the same thing, then kubernetes managed the containe orchestration. it was fun.
ggm 6 hours ago||
NFS diskless is the more common approach I've used but this is very cool.
KaiserPro 4 hours ago||
NFS diskless was easier for me to setup when I was doing it.

THe caveat was, you needed readonly root, so that meant freezing the OS, anything that needed changing was either stored in a ram disk (that you need to setup) or a per host nfs area (kinda like overlayfs, but not)

yjftsjthsd-h 4 hours ago||
Why would you need a read-only root? Do you mean to share across multiple machines?
KaiserPro 28 minutes ago||
Yeah it makes things a bit easier to debug. Originally my system was designed to run on multiple machines at once.

If you needed to update the root dir, you chrooted into it and did the (yum) update.

ahepp 5 hours ago||
When I tried root-on-nfs I had a lot of issues. The Redhat and Arch package managers don't seem to like it (presumably a sqlite thing?).
contingencies 5 hours ago||
You can download the rootfs, extract it to a ramdisk, and just run in memory. This is fast for everything. Unfortunately, memory just got super expensive. Fortunately, Linux requires ~no memory to do many useful things.
a96 2 hours ago|
Very neat. I've never seen the Debian installer using iscsi. No other installer either for that matter.

Looks like ZFS is only used to store the image on the server, though. I was expecting this to be more interesting because of that.

More comments...