Top
Best
New

Posted by blobmty 6 days ago

DAG Workflow Engine(github.com)
70 points | 55 commentspage 3
esafak 5 days ago|
I don't see any references to existing orchestrators, which are way more complete, so I presume you did this as an exercise?

Just seeing YAML used for workflows in this age makes me automatically nope out.

afshinmeh 5 days ago|
Curious, what format would you prefer to use to represent a workflow instead of YAML?
esafak 5 days ago||
Type-safe code. Workflows are not configuration! If I wanted YAML hell I could stick to Github Actions.

But that's only the start. There are a lot of other things I would expect of a new workflow orchestrator in 2026 so if you are not comparing yourself to the competition you probably don't know what you're getting yourself into.

afshinmeh 5 days ago||
Yeah, that makes sense. I looked at a few workflow orchestrators and I'm building something that I will release soon, but my thinking is that the "workflow engine" should be an abstraction that takes the input and executes the steps. "What" you use to define that workflow is probably the SDK layer though, but I can certainly see the value in using type safe code to define as opposed to a YAML file.

I'm mainly focusing on the portability aspect of it (e.g. use TS/Python/etc. to define the workflow/steps or just simple a simple YAML file).

verdverm 5 days ago|||
Are you planning to map those varied definitions onto varied orchestrators?
afshinmeh 5 days ago||
Sort of. My thinking is that the input to define the workflow should be anything you prefer to use (TS, Go, YAML, etc.) and the orchestrator's job is to model that and execute the job, given your deployment model.
verdverm 5 days ago||
There are a number of widely used orchestrators, it would be nice to deploy to one of those vs a new kid on the block
afshinmeh 5 days ago||
I'm mainly looking at Rust based projects and haven't been able to find something to use out of the box, without hacky RPC/Shell execs. Curious if you have any suggestions?
verdverm 4 days ago||
The big data world largely revolves around python, like much of the AI world. Many of the people are more focused on the science than programming, so they aren't interested in the same arguments we often see about rise being a good choice for implementation. They want to use a language they know well to get their job done, hence something like Airflow be asked about in other comments.
esafak 5 days ago|||
[dead]
535188B17C93743 5 days ago|
[dead]