Posted by blazarquasar 1 day ago
https://github.com/agent-substrate/substrate
(For context I built something very similar to this the past 2 weeks for my homelab, trying to solve many of these problems. This comment is an edited version of an unreleased blog post I wrote last week.)
- Run code in secure microVMs or gVisor. Docker is not good enough. Qemu is not good enough. A secure environment for running untrusted code is the bare minimum. I don't see Firecracker in the repo yet, but that's ok the idea is there.
- Fast resumption. In my homelab, time-to-first-message is around 11-12 seconds. That's half setting up the pod, and half resuming the CLI (e.g. `codex resume ..`). Why resuming? In my homelab agents are commonly blocked waiting for CI or waiting for me to approve an action, in this case I stop their container to keep resource usage low. Then for resumption, you definitely don't want to waste the agents time by giving a new ephemeral disk and forcing them to re-clone and re-build. For microVMs this is not actually straightforward, for example Firecracker only allows block devices, so re-attaching an agents disk workspace requires a custom storage interface
- Zero Trust. Codex CLI permissions for example are extremely broken. "Can I run this 500 line long command? or allow any command starting with first 100 chars always?" More reasonable grants are needed.
I don't understand yet how they will surface Zero Trust notifications. In my homelab it's a Forgejo comment linking to an auth service, and a ntfy.sh iOS notification which opens up the auth service.
I don't get why they to restore the RAM of the agent env. Maybe to fully optimize resumption. Idk, I don't have that much RAM in my homelab, my agents use a ton, testing stuff in Chromium making screenshots for me. I can't keep RAM for 100 workspaces from the past 24 hours in RAM.
MITM gateway is very cool.
I'm curious how they will integrate with microVMs. I just wrote yesterday[1] about how there are NO GOOD OPTIONS for this atm. Kata is decent but the attack surface it introduces makes me uncomfortable.
[1]: https://srcreigh.ca/posts/auditable-kata/
But anyway, even if this project is abandoned out of the gate by Google, we should be happy, it sets the bar where it should be. I'm excited to learn how they solved these problems differently than I did.
But for most things, I find resuming with memory is more trouble than it's worth. If you always resume from memory, you lose the ability to control the state of a VM. It's much easier to define which services should run than to define which active RAM state should be purged
Similar to why "did you try turning it on and off again?" is good for system reliability.
This is going to be decomposed. I believe the plan is to offer resumption with disk state only as well (and the RAM snapshots will need to be discarded from time to time, if you update the underlying code of the agent, or switch CPU types).
By the way, the RAM snapshots are not kept in RAM, they are serialized to disk, or uploaded to object storage.
For further isolation, I like to use nono inside a smolvm instance.
I wouldn’t dismiss smolvm so fast. It brings together many ideas that make the whole very interesting.
https://blog.trailofbits.com/2026/08/26/vms-wont-contain-cyb...
Why? Firecracker mounts very few host systems into the VM, exposing minimal host code to malicious guests. Qemu and smolvm expose much more.
So yeah, smolvm is more like a docker or qemu alternative, definitely useful but NOT relevant to the discussion of sandboxing malicious code
It’s better than Docker, but it can’t be compared to Firecracker at all. Firecracker actually minimizes the attack surface whereas smolvm does not
I think Scion has so much more mature a disosition: you could write OpenCode plugins that enhance the runner, and use that locally, and use it in Scion. With Ax/Agent Substrate, you are opting in to a pretty huge stack that is just Agent Substrate, that is their runners, their harness, their substrate. I do think their actor model is pretty neat! It's neat having the agent have such primacy! But it feels so much less integrative, is such it's own thing. Scion, to me, is much more interesting an effort, that similarly helps scale out agentic workloads.
The website makes me think the contrary: It is described as “low opinion” and explicitly mentions that the running tasks don’t even have to be AI agents. Can you explain in what ways you’re more locked in than the website suggests?
Scion at the same time talks much more about concrete agents, giving me the opposite initial impression.
> You need a Kubernetes cluster, ko (brew install ko), a container registry your cluster can pull from, and a reachable Agent Substrate Control API (in-cluster default: api.ate-system.svc.cluster.local:443).
> make deploy AX_IMAGE_REPO=<your-registry>
> This deploys Redis, then builds and deploys the control plane images with ko. Everything lands in the ax-system namespace.
What's nice about Scion is that it runs existing systems. It runs Claude, it runs Code, it runs Pi, it runs OpenCode. By contrast, "low opinion" means build something new, from scratch, atop this brand new platform.
Note that both of these are designed to work at some scale. Agent Substrate specifically is somewhat coupled to Kubernetes, is my impression, but honestly that's fine with me. Scion can run on Docker, Podman, Apple Container, Kubernetes, or Cloud Run. It's good that we be able to run these relatively quickly, but (especially with LLM assistance) the idea of running some substantial dependencies / services to run these things does not seem like a bad thing. If anything, I'd prefer having some well known services underfoot to these all being recreated afresh.