Posted by icyfox 3 days ago
I fell in love with durable execution a few years ago, after working at a company with a heavy pubsub architecture. In comparison to job orchestrators that force a certain task design or special syntax, many of these durable libs lets you write business logic in regular code: `asyncio.gather` to parallelize tasks, `for` loops, conditionals, and let the runtime figure out how to run them efficiently across a cluster of nodes. But most of the current solutions require a heavy broker, runtime hacks to try to ensure deterministic workflow logic, and feel like you have to learn a whole new ecosystem just to get started.
A few unique design decisions in waymark:
— Write jobs in normal code (right now Python but JS coming soon). Instead of running control flow as-is, which requires booting up an interpreter to replay past state, we’ll just compile your code directly. We walk your AST and map your logic to our stackless VM bytecode. We support variables, conditionals, await, sleep, helper functions, loops... basically what you’d expect in workflow logic.
— Since we’re compiling a language subset, we can proactively flag errors in your workflows locally instead of when they run in production.
— It only requires Postgres as the task broker. We’ve spent a lot of time tuning our design to play nicely with its strengths and limitations; our current benchmarks show around 1500 workflows/second but we still have some more optimization work to do here.
— Actions plug-and-play with FastAPI’s dependency injection syntax. So if you already have an ecosystem based on typehinting/DI, it should fit like a glove.
It allows for parallelization of workflows across one machine or a whole cluster. The bulk of the work per node is done by our central Rust core - which distributes workflows to properly saturate all cores. The idea is to be as batteries included as possible with the broker, webapp, and runtime.
What to use it for? It works great as the runner loop controlling agents, but it also works well for basically any task that you need to kick off in the background. It’s largely replaced our way of writing api code. Most frontend actions at this point just push to a simple REST endpoint that kick off a background job.
I spent an embarrassing amount of time making our https://waymark.sh site interactive - it’s probably the best place to learn more. We’re starting with Python support and looking for community input on what language to ship next. Would love to hear your thoughts!
- we're "Bazel for data"
- but...we have no Starlark. We're trying to port Airflow but it's not playing well with the way we cache partial results.
Amazing to me that I'm only now seeing the architecture pattern of "pure coordination language connecting sandboxed imperative components", which is used by Nix and Bazel, applied to distributed workflows. If somebody applied this to batch data processing you might have a Pachyderm that actually works better than ours did.
In principle I would love to have something like this working for arbitrarily large ETL flows; ie. write the logic you want and it tries to figure out a good distributed algorithm to get the outputs. At the moment flowing data through the cluster is the biggest bottleneck. We generally assume that action inputs & outputs themselves are pretty small at the boundary even if their containerized context is quite large.
Where did you hit the most snags with caching?
To give a somewhat longer description of what Pachyderm does/did:
1. Connect data-scale version control with pipeline orchestrator.
2. Version control tells orchestrator which shards have changed since the last run and which ones haven't.
3. Orchestrator only processes changed shards from one run to the next, instead of all of them.
4. Profit.
That's fine for e.g. one mapreduce, but,
1. If an action's behavior changes, you can't re-use its prior results (usually—sometimes you're just adding print statements to debug a failing shard)
2. If the overall pipeline changes, you can't re-use results from the first change onward
3. Figuring out if the overall pipeline changed is not necessarily easy, and if you're always throwing out everything just in case, your product is not helping people.
So, the way Waymark uses a compiler for the orchestration layer would've worked so well with our model.
It's also just satisfying—once I learned in detail how Bazel/Nix work, I see a fit for that architecture pattern in anything remotely pipeline-y, and it drove me nuts that Airflow/Beam/Dataflow weren't really using it. So I'm very happy to see this for that reason as well. And, as if that weren't enough, it's so cool that it introduces a pure language at the coordination layer (which this type of architecture basically needs, hence the Nix language/Starlark) without actually requiring a new language, or even a new file. Amazing!
The main issue so far is that our VM that does the workflow orchestration and the actions runtime are living in separate processes - but we even then, we could do tricks to share memory between them for more efficient / zero-copy data passing... That's not something we have right now though.
With Pachyderm, part of our model was that the "actions" were arbitrary, user-provided containers[^1], and the inputs/outputs were mounted filesystems (`/pfs/my_first_input` was a directory containing the first input dataset, or one shard of it, etc, and `/pfs/out` was a writeable directory where the output for that shard would go[^2]). We built the (versioned) filesystem that data in /pfs came from/went to, but you could probably use ceph or something if you're not trying to do versioning. That's how we dealt with big inputs/outputs.
We never tried to optimize the single-machine case, for better or worse.
[^1]: Using arbitrary containers was a good choice for us, though, because we were marketing to spark users and a lot of them were frustrated with how hard worker-dependency management is in spark. So we got to say "put perl, weird sequence analysis libraries, whatever you want, in a container, and if it can read and write files, we'll mapreduce it over your data".
[^2]: `/pfs` was a terrible choice because it's nonstandard. Really wish we'd used `/mnt` or `/mnt/pfs` instead)
With agentic workloads, though—especially trending towards high network calls and smaller models—I do find a lot of concurrency to be a pain in the ass to pick up where we left off. Is that what "durability" ameliorates?
I just wonder what I'm leaving on the table.
What this means is when you have an async fn that you spawn in a loop where each iteration also spawns a ton of async fns which in turn call actions - there's no separate bookkeeping to do parallel to the execution itself. No replays (unlike other tools like Temporal), meaning the thing just goes, as-is, and when it crashes it can load the state and resume from where it left off - without the need to match the state of the interpreter to the persisted state of execution graph. We just store the real, actual runtime state.
- Each LLM call and tool call becomes a checkpoint. Without durable execution you're pretty much stuck doing db saves/restores if you detect that this agent might have a previous context window. Agent frameworks are fundamentally just harnesses around the underlying model: you `while loop` until some server gives you some payload back, you add that to your context window list, and you keep going until it terminates. Durable execution moves that loop into the workflow body itself (a quick example here https://www.waymark.sh/python/tutorials/building-a-from-scra...).
- Long waits. Sometimes you do want your agent to wait for an arbitrary amount of time (ie. wait for this compilation job to finish - look again in 5mins, or maybe 5 hours). Durable execution makes sure that sleep timer will wake up regardless of what happens in the system state.
- Easier monitorability. Instead of having to sift through interleaved logs to figure out what was going on, you can see every large function call with a clean input/output contract. Useful especially in tracing down raised Exceptions and the data flow that it took to get there. Good for agents and regular code alike.
Of course, if your system is largely idempotent or your agents are mostly classifiers that are working in short internals - a simple db save/restore can work perfectly well too.
I like this one, as in the context of workloads like these, going with anything more complex than a Postgres is overkill.
Great to see some experimentation in this space. I really like the idea of 'make this async function a background process' using a simple annotation.
What kind of IR do you use? I often wonder what a WASM based durable execution engine would look like. And what that pattern would unlock.
We have two things, both custom: bytecode (what our VM uses) and IR (what Python is parsed into); there's no stable ABI yet since we practically don't need it, yet, IR is specified as a protobuf schema and bytecode is a specified via rust-native types in our framework for building instruction sets.
In principle, where WASM would go is it would be our compilation target instead of our own bytecode. We've thought about using WASM and, independent of that, about using LLVM compiler facilities. They don't have to go together, and, arguably, tapping into LLVM would be more interesting than WASM (I'm mentioning them together one of the intuitive perks from using WASM would seemingly be using WASM compilers too).
The thing is, we would, most likely, not have too much benefits from using LLVM or existing compilers from various languages into WASM; we compile things in a slightly different manner, and, I suspect, it would be possible to re-use some of the existing optimizations for control graph - but not the low-level ones.
As for WASM - we use a stackless VM, and WASM has an execution stack as a requirement - so we can't seemingly use WASM. That said, in practice we could use the same technique rust uses for compiling futures, and run on a WASM VM with a single stack frame holding our state machine and just doing everything in it. That's pretty close conceptually to how we have implemented in our custom VM now.
I find this really common on HN - someone posts a project that addresses a problem and all the intro and docs assume you understand the problem and couch the discussion in somewhat domain specific language.
It might be interesting to me. It might even be relevant to me and I don't know it yet. The problem is I need to work really hard to get to the point of understanding that.
This is an appeal to authors of such posts: please give a motivation that provides the necessary background to understand the problem, and ideally do so in a way that a technical but non specialist reader can understand.
> This is an appeal to authors of such posts: please give a motivation that provides the necessary background to understand the problem, and ideally do so in a way that a technical but non specialist reader can understand.
I agree with this. For Show HN posts, this would really help broaden the audience as well as lift up the audience’s knowledge and enthusiasm.
Consider a microservices architecture (for the sake of example), where a user sign up needs to create an account in three services: auth-svc, avatars-svc and emails-svc.
Your options to do it are either an event-sourcing architecture where your services have to orchestrate together via somewhat-awkward message-passing, or write another reconciling service where you write the logic more-or-less directly, but with a ton of code-overhead to make the execution reliable.
Waymark would help with the second approach, because it soaks in all the durable execution complexity, and gives you a way to just write the logic code and handle the edge cases - with a fraction of the code and systems you'd need otherwise.
I'd even dare to say - a tool like waymark, the second approach is pretty much impractical altogether, hence why the event sourcing microserivce architectures dominate today.
But, yet again, that's just one example of where waymark would be useful. If you can write the code where you can split the orchestration logic from the execution of the individual side effects - then with waymark your orchestration code can be just the logic you need, without the extra steps to persist the progress - thus making it easier to reason and mental model simpler.
Similar in what way:
- Tower allows you to compose the calls with, essentially, helpers, and express your decisions on how to apply retries and timeouts concisely.
- Waymark also allows you to express your decisions on retries and timeouts via compact declarations.
The difference is, though, that with tower you provide the execution, while with waymark the execution is handled by waymark itself. So, this becomes more like tower+tokio::spawn, except this is not even the key thing waymark does.
The main feature is, speaking in rust analogy terms, that we convert workflow into a future, and not just any future, but a future that can be serialized and stored in postgres; then we run it to completion, and if anything crashes we can resume on a new node from the state that was last persisted.
Persistence is automatic and happens on any "external input" that the internal VM absorbs (typically the result of a side effect like an action execution).
So, while tower can, in principle, add persistency to your call (and does handle timeouts and retries already) - it doesn't handle the orchestration of the whole program in the same way waymark does.
One more difference: waymark is written in rust, but you don't write in rust (yet) when using it. You have to write the code in Python (now) and we'll be adding more supported languages later. Rust is obviously on our list too - but we'd want to prioritize based on the demand.
Here's the code for the workflows we use in benches:
https://github.com/piercefreeman/waymark/blob/main/crates/li...
Hugging face had issues with temporal?
Workflow definitions looks similar to Temporal, by design. They want you to program in your native code and separate actions at boundaries - as do we. A few fundamental differences -
1. Temporal gets its durable execution properties by booting up a full interpreter (Python/JS/etc) just to replay the previous state of your workflow. This requires your full logic to be deterministic because it's actually re-serializing variables from the cached state of the action outputs.
Waymark by contrast does everything by compiling your run definitions. This allows your logic to be nondeterministic as long as it implements the supported control flow. This compiled representation is also executed in our Rust VM core, so it's easily parallelizable to your whole machine.
As an aside - the Temporal maintainers also have to re-implement async handling in each of their supported languages. They have a rust core iirc but because they actually hook into the native language, they need to implement their own async abstraction. There are a lot of footguns with these implementations that they need careful handling to avoid. Since we share the same workflow VM runner against all languages we anticipate being able to add language support and keep it maintained much easier.
2. Due to this compilation, it's easier to catch errors locally and in CI before it reaches production. There's no notion of 'workflow changed' / incompatibility errors with future waymark workflows.
3. We have a simpler hosting paradigm. Instead of needing a full broker that you need to scale separately, we use Postgres as the only external service required. You can leverage your existing database expertise instead of needing to move all the knobs and dials of Temporal/RabbitMQ/etc.