Top
Best
New

Posted by cbrake 5 days ago

Is it time for a new Embedded Linux build system?(yoebuild.org)
79 points | 74 comments
KingMachiavelli 1 day ago|
I've been using Nix for "embedded" systems for few years now and it works fine. I don't quite understand why "embedded" has historically meant learning a completely separate tool. I've been building for x86 and ARM servers for 5+ years now, so why should targeting an ARM board be any different?

Like the article points out, the software stacks people use for embedded devices are the same as people use non-embedded use; Rust, Go, NodeJS, and sure still C++. The only real difference with embedded devices is non-OS components like the bootloader (u-boot, EDK2) and customizing the device tree. (And of course firmware flashing). Fundamentally those are all just packages that I can describe in Nix. I don't need a separate tool just because the board is small.

IMO the embedded space, especially in the US, is already pretty Niche. Most companies either just ship the vendors BSP example (Ubuntu/Debian/Yocto) and pay very little attention to the detail or re-useablity. Once you vendor declares the BSP EOL you are stuck unless you re-implement it yourself.

Using Nix (or Guix) has the massive advantage of a large and active community that isn't fractured like the Yoctoo/Buildroot community. (By fractured I mean there may by many, many people using those tools at $DAYJOB but due to vendor BSP customization they likely share much less with the upstream community maintained sources).

saidinesh5 1 day ago||
> I don't quite understand why "embedded" has historically meant learning a completely separate tool. I've been building for x86 and ARM servers for 5+ years now, so why should targeting an ARM board be any different?

Because embedded usually means dealing with complete functionality provided by the soc, bare minimum for the init system to get quick boot times.

If you just want to boot the CPU and run some IoT app, device tree + kernel + uboot should be enough.

But for anything with a GUI, You'd probably need the vendor provided binary blobs for GPU accelerated UI, proper gstreamer packages with all the patches to make it work with the binary blobs... properly packaged Qt etc ... Not to mention read only rootfs and A/B partitions for upgrades to deal with power loss headaches.

I haven't used Nix for any of these but most of these things are available ready to use with the vendor provided buildroot or yocto setups. And it would've been just more work to get this all working with yet another build system/package manager

jo-m 23 hours ago|||
I was thinking the same.

However, reading through the docs, the author clearly knows about Nix/Nixpks and has factored some of its properties into the design: https://docs.yoebuild.org/nix.html

KingMachiavelli 12 hours ago||
Actually, quite an honest and in-depth review with many genuine good points about the state of Nix. If you genuinely need sub 1GB image sizes, you would have to reimplement many of the Nixpkgs conventions like glibc/systemd.

I’d still probably just implement this in Nix/Nixpkgs so I could leverage the rest of the ecosystem but that’s not trivial at all.

I wouldn’t even say Nix as a language is better than Starlark. The author is certainly well informed in the choices made.

Palomides 1 day ago|||
I'm not very familiar with nix, how well does it do with cross compiling? is anyone actually using it for yocto sorts of domains?
sollniss 1 day ago|||
You can generally remotely build for any supported architecture[1]. The build process will be painfully slow though if you have cache misses.

[1] https://search.nixos.org/options?channel=26.05&query=boot.bi...

throwawayqqq11 20 hours ago|||
Nitpick: binfmt.emulatedSystems is not true cross compilation. It sets up a nixos to enable emulated native compilation. After painful tinkering, i prefer this way too. When i recall correctly, for cross compilation you'd have to use nixpkgs flags.
actionfromafar 19 hours ago||
At some point it's a matter of perspective. If the emulated system is fast, it feels the same as "true" cross compilation. In the end it's a bunch of bits and bytes which produces some other bits and bytes when we poke it. If it goes fast, it's good. If it goes slow, it's painful.
nine_k 1 day ago||||
We do Nix-based cross-compiling for non-embedded software (ARM vs x64). I'd say it works mostly transparently.
rendaw 1 day ago||||
Nixos can cross build os images for Le Potato with device tree stuff: https://hub.libre.computer/t/nixos-for-aml-s905x-cc-le-potat...
alright2565 1 day ago|||
[dead]
jimktrains2 1 day ago|||
> I don't quite understand why "embedded" has historically meant learning a completely separate tool.

Because historically, and I would argue should still, refer to very limited hardware, in terms of memory, processing power, and energy.

zamadatix 1 day ago||
If you're sticking the Linux kernel into something you've already past the scale this should require a separate build system to accomplish.
Brian_K_White 1 day ago||
I guess this was just so true everyone found it uninteresting like saying water is wet eh?
superxpro12 15 hours ago|||
Hey, you completely forgot about embedded devices that dont run linux. It's not all linux ya know
nrdvana 1 day ago||
I share some of the same observations that seem to have motivated this project, but my solution was to just use Alpine on the same architecture as the target (possibly via qemu) and then export a subset of it to a filesystem image. I implemented it as a perl module collection with no dependencies other than core perl modules, and then run the export from within the image itself. Among other things, it lets me use strace to find the runtime library dependencies of the things that need to be in the image.

https://metacpan.org/pod/Sys::Export

dima55 1 day ago||
Modern SBCs are just normal computers and not "embedded" in the traditional sense. You can generally just use Debian, and spend time on the actual project, instead of wrestling with the system
Gigachad 1 day ago||
They aren't normal computers in the same way x86 is. They usually need proprietary drivers in custom boot processes, forked kernels with patches, etc. I'm yet to see a single SBC I can just download the iso from the debian website and get going. You always need some custom build.
dima55 1 day ago||
ARM boards need a custom kernel and bootloader. These aren't things managed by the distro. The userspace IS managed by the distro and IS standard. Using yocto to manage userspace maybe made sense 15 years ago, but it has long since become far more trouble than its worth. Debian supports most architectures out of the box, has good cross-building infrastructure, many thousands of ready-to-use packages, and is non-weird.
cozzyd 1 day ago|||
Debian handles userspace fine but there is a lot of image customization needed a lot of the time. I have experience with developing images for a beaglebone black, where I forked the bespoke image builder (to create user accounts, mount points, set up udev rules, device tree overlays, ssh keys, etc) and that was a pain to maintain. I suppose I could have create debian packages for some of those things, but I would still have needed to set up repositories. Now I'm using an SoM with Yocto and while the learning curve is quite a bit steeper, it is much easier to customize for our experiment's needs. Also, AI assistants are really good at spitting out yocto recipes (that usually don't work immediately, but gets you 90% of the way there).
menaerus 23 hours ago||||
You're missing the point that when you're building the product, and not using SBC just for fun, the image you are building, and yes even the userspace, is highly customized. Running a full-blown Linux distro is a non-requirement, and often incorrect.
j4k0bfr 1 day ago|||
For more mainstream SBCs, totally agree. However a lot of the niche industry boards (especially new ones) only support Yocto out of the gate. Debian might be "supported" by the SoM/CPU vendor but manually reapplying kernel patches and praying it boots is a PITA.
Const-me 1 day ago|||
I agree. In the past, I have successfully used Debian and Alpine for embedded. Never needed to compile OS kernels or standard DLLs, other people already did and published in these package repositories.
nottorp 22 hours ago||
Except you may have just a 512 M industrial sd card to run on. Or even less.
wasting_time 1 day ago||
I'm surprised to see no mention of Guix. It solves all of these issues, and already has a good story both for cross and native builds on a variety of architectures.

Adding new targets is deceptively easy, just copy an existing template and substitute your values.

https://codeberg.org/guix/guix/src/branch/master/gnu/system/...

https://codeberg.org/guix/guix/src/branch/master/gnu/bootloa...

yjftsjthsd-h 1 day ago|
Does guix have any advantage here over nix?
wasting_time 1 day ago|||
I didn't know Nix had support for embedded systems. Where are the images defined?
KingMachiavelli 1 day ago||
What do you mean by image? In Nix you instantiate a NixOS system and specify the architecture. There are a number of tools to build a disk image from a NixOS system.

$ my-arm-system = nixpkgs.lib.nixosSystem { system = "aarch64-linux"; modules = [ ./configuration.nix ]; };

$ :b arm-system.config.system.build.images.iso

throwawayqqq11 20 hours ago||
PSA:

nixos-rebuild build-vm (--flake DIR#HOSTNAME / configuration.nix)

./result/bin/SCRIPT

to build and launch a nixos vm with qemu.

KingMachiavelli 12 hours ago||
IMO the nixos-rebuild scripts are unnecessary and just obscures what’s being built.

build-vm is the same as building the configs “config.system.build.vm” attribute.

nixos-rebuild is fine for getting started but you end up needing to use the attribute path to do anything more complicated like CI caching or inspecting the drvPath.

mghackerlady 1 day ago|||
Lisp
lukeh 1 day ago||
I haven't used it, but there's also Avocado OS [1]. Myself, I use Yocto for our embedded build. It's not perfect but for our application, which involves building a bunch of software that uses both Swift and Flutter for an embedded audio product running on a RPi CM4, it works just fine. Shout out to Mender for A/B updates.

[1] https://www.peridio.com/avocado-os

hparadiz 1 day ago||
Gentoo is literally designed for this but people keep coming up with other non-solutions. A Gentoo stage 3 with a custom kernel can run on a potato.
palata 20 hours ago|
Doesn't Yocto come from Gentoo to some extent?
lpribis 17 hours ago||
Bitbake (Yocto's build system) started originally as a fork of portage (gentoo package manager), but they have diverged a ton. Very similar concepts though.
ValdikSS 15 hours ago||
Wow, no cross-compilation as a feature?!

Architecture emulation with qemu-userspace is 5-10x slower than running a native build.

The only 'proper' no-friction cross-compilation system I know is Debian/Ubuntu, because it allows to install foreign architecture libraries into your native system, and has all the wrappers for cross-compilation in dpkg-buildpackage.

This way you don't have to maintain cross-compiler or toolchain/sysroot, you just install whatever dependencies you need with the regular `apt install` of another architecture.

I'd like to have more friendly tooling around that instead of a slow compilation of architecture emulation.

okanat 1 day ago||
The problems building with Linux and GNU environments exist because they were terribly designed with assumptions like

- You're building on the same native system as GNU and Linux packages, you install them globally in the same places that servers and desktops use

- Your C, C++ compiler and entire toolchain and other binary utilities with the kernel is a one single unit that you can only change one part at a time

- You use the same up to date headers with glibc, gcc and Linux kernel

- You're building software in the same universe of all the other packages, especially gcc libraries (libgcc_s, libstdc++), glibc (especially bad since ld-linux.so is part of it)

- The build system only uses standard paths

The reason Yocto is so complicated is that developing in a Linux environment actually sucks when you're not writing web-oriented or server / VM software. Yocto fixes it. It introduces a good set of abstractions that work around terrible design decisions that were made in overall Linux ecosystem. There are a lot because the OS design is fundamentally broken, especially with C-based toolchains which is 99.999% of the ecosystem. Current C toolchains including MSVC strongly ties OS with the C's internal types and bad decisions of 70s.

As always all articles whose title asks a question are answered with NO, 99% time. By taking away the cross-compiling abilities and the workarounds doesn't fix the brokenness of Linux and overall FOSS ecosystem.

If you're looking for how a better embedded environment looks like, look at Rust toolchains. For Linux take a look at musl-libc based ones (you 100% need a systemd distro to get away from nss complexities that musl introduces). Or even better take a look at relibc. There are barely any assumptions about the target filesystem and tooling in Rust toolchains, unlike C/C++/Make toolchains. There is redox OS but it is still in slow development and they stuck with Make, which I think was a bad decision. Android uses its own build tooling but cannot run away from C/C++ tooling unless Google revives Fuschia.

sn 1 day ago||
This sounds like fun, but I'm not sure how changing build systems fixes the vendor problem. They will just ship their proprietary blobs built on top of this new build system instead of the old one, and will not change other behavior. Buying from vendors who upstream or minimally open source their work might help.

Yes, skipping builds from source can be faster, and, I don't know that you need to throw away yocto to do that https://rootcommit.com/pub/conferences/2024/elce/yocto-binar...

From the statement about "Caches builds so no piece of software is built twice." I'm guessing that the author has not enabled sstate caching https://docs.yoctoproject.org/dev/overview-manual/concepts.h...

I'm a little doubtful on the post's assertion that yocto is only needed for "deeply embedded regulated products." For products that have to follow the https://en.wikipedia.org/wiki/Cyber_Resilience_Act coming up, they will almost certainly need:

* SBOM management https://docs.yoctoproject.org/next/dev-manual/sbom.html

* CVE (or its successor) tracking https://docs.yoctoproject.org/dev/security-manual/vulnerabil...

* License management https://docs.yoctoproject.org/next/dev-manual/licenses.html

And I'll echo what some other folks have said here: if you don't need features like that, bundle your custom bootloaders and kernel with a binary distribution like Debian and call it a day.

itomato 16 hours ago|
Yes, and if they have done the hard work to deliver features in the kernel that provide a moat, that blob is the hardware.

Reversing and reimplementing the closed bits are what I want my Buildroot pipelines to do.

That’s my use case though; targeting COTS devices with (mainline) Linux support that don’t do all they could.

I have done it the ugly way (in production) and I want elegance in my old age.

klysm 1 day ago|
I just use Debian these days
More comments...