Posted by cbrake 5 days ago
yoebuild does seem interesting at least as a proof-of-concept.
When someone starts with "why cross-compiling the system if it's bearable to compile it directly on the device?", it really sounds like they are just not comfortable with cross-compilation. To me it would be similar to say "why would you need a compiled language, with this weird compilation step that nobody understands, when you can just write in an interpreted language?". Except that developers wouldn't generally proudly admit that they don't know how to use a compiler.
A valid reason I see to require a new embedded linux build system would be if the existing ones (mostly buildroot and Yocto) were unable to build a modern OS (e.g. by not being able to integrate some security features). But I don't think that's the case. This article sounds more like very verbose marketing for a system claiming that it can do similarly without having to learn as much (which I honestly doubt).
To me it's often the same pattern:
* Meson is nice, but it doesn't remove the need to use CMake, and it's not fundamentally simpler or more powerful (it's more comfortable at most).
* Jujutsu is nice, but it doesn't remove the need to learn git, and it's not fundamentally simpler or more powerful.
* Yoebuild sounds like even if it got traction like Meson and Jujutsu, it wouldn't remove the need to interact with Yocto for embedded linux engineers.
As an embedded linux engineer, if anyway I have to interact with Yocto and therefore learn Yocto, why would I learn another one that isn't fundamentally better? Usually people who go for those "simpler" (at least on the surface) tools do it because they don't want to learn Yocto. And for an embedded linux engineer, isn't it a red flag to not want to learn the most famous embedded linux build system?
It reminds me of Balena/Resin, which seems like it's targetting people who are not embedded linux engineers, don't want to become embedded linux engineers, but still want to distribute an embedded linux. To me that's generally a red flag.
I agree this is true a lot of the time, speaking as someone who has made many Yocto messes and later regretted. But you also can't discount all of the messes made by silicon OEMs that you are forced to deal with to use their SoCs.
E.g. I recently did a project with Yocto on a Zynq US+. Look at the meta-xilinx layerset (or, God forbid, petalinux) and tell me they haven't made a complete mess of it.
The complexity and lack of upstreaming in kernel and userspace for so many of these SoCs, coupled with non standard boot process and devicetrees means you will be forced to use these terrible OEM layers. I don't think a new build system can fix this at all, it's a cultural/commitment that needs to change from the OEMs.
At the end of the day, of you are doing any complex embedded Linux work as a small team, you must use Yocto because it's what the OEM supports. I don't have time or resource to debug DFX not working because some userspace tool was built with slightly different flags than expected by the flaky ass xilinx upstream.
Yes, totally! It's a bit sad because we don't usually have the luxury to select the hardware based on the quality of the BSP. First because they all seem to be terrible, and second because it is somehow "hidden" cost. One selects the hardware based on its price and capabilities, and the software part is left as an exercise.
But as you say it's not a problem of Yocto, it's a problem of the OEMs being in a position to make a mess because it doesn't have an impact on them. A bit like Apple is terrible for developers (in my experience) because anyway developers don't have a choice.
You can make great things with that, but it was easy to be "holding it wrong".
I still recommend CMake at work, it's the least offensive solution in most situations.
Probably CMake is not a good fit for very, very big projects. But the vast majority of projects are not that big.
Usually what I see with CMake is self-inflicted pain, often where the devs start adding custom functions and mix it with python and stuff. If you keep it simple, it's very clear and does the job with surprisingly few keywords, I find. And I have setup tens of projects with CMake.
My impression in recent years is that these image cross build environments are just not as frequently needed as they were back in the day of their invention.
My most recent embedded linux environments were just embedded archlinux.
No need to cross build an image, just install and run the minimized linux environment right on the target.
Of course, a big part of the need for these cross-tools is that it seems most modern embedded linux developers are running windoze on their development workstations 8-/
At the end of the day you need some cross compilation just for board bring up.
If you're playing with some platform for which this has already been done, then sure, but that's not really the "normal" way of doing embedded.
The main reason to not run Debian is that Debian usually makes lots of compile-time decisions for you, and chooses to maximize the enabled options of the software. Alpine makes the opposite choices for you, creating very minimal feature sets.
It's been working well for me, but my use cases is a lot more like a custom in-house DDWRT firmware kind of a thing. If you've experienced additional constraints that make this pattern unworkable, I'd be interested to hear about them.
My preference is to cross build packages for each application on my workstation, and maintain a local package repository from which individual packages are installed on the target.
I find archlinux extremely useful in this configuration, because even if I'm not building on the target, the same package maintenance and other system commands are identical to my development workstation.
Additionally, archlinux provides a well documented, and fairly simple process for hosting your own local repositories.
Instead of cross-compiling with a large meta-layer stack, the tool builds kernel, rootfs, and applications together using one engine, with a CLI, TUI, and AI assistant talking to the same core. All you need is the tool, Docker, and Git — no global SDKs or hidden state.
It’s pre-1.0 and rough around the edges; I’m sharing it early to get feedback from people who live in Yocto/OpenEmbedded, Buildroot, Nix, etc. I’d love to hear where this breaks on your boards, what workflows feel wrong, and whether the “native builds + AI-aware build graph” direction seems promising.
This is the complete opposite way, actually.
We need cross-compiling that is just as effortless as native compilation.
You should be able to build complex software on a powerful computer and perform costly optimization, then run it on a low-powered device.
Even if you don't have a beefy server of the same architecture, you can probably run it in qemu instead of docker to the same effect. And even if qemu is slow, you can run a build of the kernel and your in-house stuff in parallel on 64 cores and not really be affected by the qemu slowdown.
I'm interested to hear counterexamples, though.
What if this hegemony in ISAs is partially because of the high costs of porting software and toolchains?
Maybe the next generation of embedded devices will have ISAs that are, say, extremely optimized for real time industrial controllers, but terrible at compiling C++.
It seems a little silly to give up on cross-compilation just because _today_ everything runs on ARM.
Especially because cross-compilation is not a particularly hard problem, the problem is with decades of tooling assuming the build and target environments are the same.
I had been maintaining a fork of the U of KY BDR with SBCs and Arm devices in mind, might need to revive it for RISC-V.
"Ampere processors natively support both 32 & 64 bit Android applications and require no binary translation for maximum instance density" ... "Run up to 120+ 3D cloud game instances per socket" ... "DRAM: 384GB - 512GB"
https://amperecomputing.com/solutions/arm-native
I've not personally tested this, but it very much appears that just like x86, you can use a 64-bit ARM kernel that has 32-bit support enabled to run a 32-bit userspace, likely compatible with docker, or even just in a chroot. Then with 120 cores, "make -j 120"
I’m already sitting at an expensive computer, if everything else were equal I’d rather cross compile than buy more hardware.
I’m open to reasons that’s presently too hard of a workflow to enable so I’m (literally) paying for a penalty, but I don’t see how it’s a better idea.
the various embedded build systems make cross-compiling easy at scale, which means you often don't need to think about it when adding a new package
The workflow for user space can definitely improve some of this pain but I feel like a large portion of any embedded Linux development effort still ends up in the weeds for boot related items (secure boot, proper updates, nuanced kernel patches, bootloaders, device trees, and supporting machine variants, etc). Solving those to make them easy is a hard problem for sure.
As a long time user of Bitbake and Yocto, and watcher of Yoe Linux... I kind of have to respectfully disagree with the approach.
One look at the Python or Node packaging ecosystem, and you can see how difficult trying to integrate those in a 'sane' way in embedded, with repeatability and security in mind, nevertheless wrangling something like C/C++ dependencies and native packages in a qemu-cross (or binfmt emulation) environment.
I feel like Bitbake and the wider Yocto ecosystem has essentially 'solved' cross compiling. Sure there are the incredibly complex codebases like Chromium that require lots and lots of study and patience, or esoteric compilers, etc. but for most applications I feel that especially AI tooling can write up a good basic recipe for integration with Yocto.
The unfortunate thing about AI tooling is Kernighan's law (where debugging is twice as hard as writing it the first time.) Especially in Embedded Linux, where 99.99% of the product code is written by others, trying to figure out where the AI didn't quite get it right or missed something can break things unexpectedly and impossibly for the unaware developer to fix, even breaking at runtime. So the build environment has to be simple and predictable. I think Yocto/Bitbake already strikes a good balance here, with other nice things like offline builds, repeatable builds, sstate caching across machines, PRbuild servers... things useful for big teams, so consider those as well.
Although, if someone does a 'uv'-like rewrite of Bitbake into Rust with the same feature set but faster, I am all for it...
Maybe, having Yoe or some other centralized distro make specific opinionated choices and then provide an open sstate feed, working inside Bitbake, could get a lot of the speed and customizability gains back without a big rewrite effort.
RAM is expensive now. Reasonably priced seems to be counter to that fact.
The whole article feels like a promo-tour.
> The catch is that nobody handed us the tools to keep the place running.
Software such as homebrew exists. Why is the article ignoring this? Not only does it feel like promo, it now feels like propaganda. Homebrew is also just one example; LFS/BLFS also exists for embedded linux. Perhaps the quality is less than for desktop computer systems but it exists.
> Edge devices have started behaving like cloud systems. They run increasingly complex stacks, are frequently updated, and are managed remotely over their entire life.
Companies want to become dependent on others? Because that is what cloud is all about. Top-down control. A mafia for later blackmail rack-up-the-price strategies.
> That puts the cross-compile burden squarely on embedded developers, and maintaining recipes for thousands of packages has been a steady drain on the Yocto community
Seems untrue. If you have a compiler that works, you can compile. Thousands of packages? I am tracking 3888 right now. That covers really most software by far, including python pypi addons. If a single person can do that, a team has an even easier time. The whole article seems to narrate a story that is just made up.
> Building everything from source in Yocto means long builds, heavy memory use, and powerful workstations.
No it does not. I do so every day. You need a reasonably fast computer and RAM, which is now driven up by the existing hardware mafia that must be put in jail, but it is easily possible for solo persons too. Now imagine a team.
The whole article is really really crap. Absolute garbage. The claims it makes are for the most part simply not correct. It narrates a story as if all linux developers are incompetent. I highly doubt that. It also never mentions any existing software here that would change the rhetoric such as homebrew, LFS/BLFS and so forth. It feels as if that article is just engineered to tell a "we thus need xyz". That's not good writing. I mean just look at this:
"We can leverage these change agents and rethink the world of connected products."
What does that even mean? Did AI write this? Because humans are usually smarter when it comes to opinion pieces. "connected products" - everything is now connected?
And "I've been experiment with" seems to just mean "I've been vibe coding"