Top
Best
New

Posted by tomasol 1 day ago

SQLite is all you need for durable workflows(obeli.sk)
648 points | 346 commentspage 6
EGreg 1 day ago|
Files is all you need.

https://xkcd.com/378/

tclancy 1 day ago||
Post It Notes will do if you have a good system.
contingencies 1 day ago||
Those who don't understand Unix are condemned to reinvent it, poorly. - Henry Spencer .. via https://github.com/globalcitizen/taoup
lvl155 21 hours ago||
And all you need is pen and paper to do calculations.
ChrisArchitect 1 day ago||
Related:

Building durable workflows on Postgres

https://news.ycombinator.com/item?id=48313530

momojo 22 hours ago|
Surfacing for the thread:

Armin Ronacher's "Absurd Workflows: Durable Execution With Just Postgres" https://lucumr.pocoo.org/2025/11/3/absurd-workflows/

vatsachak 20 hours ago||
Postgres doesn't cost any extra lol
faangguyindia 17 hours ago|
it does cost network stack
unnouinceput 10 hours ago||
Quote 1: "DBOS recently argued that Postgres is all you need for durable execution...SQLite is all you need."

Quote 2: "SQLite State backed up to S3". Yeah buddy, if you think S3 isn't Postgres I'll eat my foot.

orf 1 day ago||
> The caveat is that Litestream replication is asynchronous. A restore can miss the newest local writes if the SQLite volume disappears before they are copied. That is fine for many AI and experimentation workflows

In short: SQLite is not all you need, unless you’re just experimenting don’t actually care about durability, in which case you also need litestream + object storage.

Right.

gwking 1 day ago||
The suitability of Litestream for production disaster recovery is also an open question in my mind. I used 0.3.x for several years and when I tried to upgrade to the 0.5.x series there were runaway disk usage problems that would have caused downtime had they made it to prod. As far as I can tell these have not been entirely addressed, although recent bug reports suggest that they might be getting closer.

I want to love it, and I don't take open source projects like this for granted. But during my last production upgrade I chose to decommission Litestream in favor of a dumber, less granular solution using sqlite3_rsync and nightly backups because there is no point in using a backup system that is not rock solid.

0cf8612b2e1e 1 day ago|||
Postgres also does not synchronously replicate for free. You can setup both to get a confirmation write if you require that durability.
orf 1 day ago|||
> postgresql also does not synchronously replicate

By default. Generally your primary database is in a completely different failure category than a kubernetes node running an ephemeral workflow pod.

0cf8612b2e1e 1 day ago||
Either you have durable storage or you do not. SQLite and Postgres can both ensure local durability of commits. If you want distributed durability, you need to ship that data elsewhere. That is another Postgres node, object store, whatever that’s still an external dependency.
paulddraper 1 day ago|||
Not for free, but without the needing additional software.

  synchronous_commit = on
0cf8612b2e1e 1 day ago||
That’s about the local transaction, not replication. SQLite WAL also gives you strict durability.

  PRAGMA synchronous = full
bootsmann 1 day ago|||
S3 is strongly consistent, if you need it anyways you can just use s3 keys to deconflict and store the workflow state.
orf 1 day ago||
Yes, but directly using s3 as a key-value database is completely different from using SQLite + litestream.
paulddraper 1 day ago|||
"Durable workflows without the durability"

That's distributed workflows :)

faangguyindia 17 hours ago|||
try setting up replication/failover in postgres, it's much more work.
dilyevsky 1 day ago||
i mean it's durable as long as nothing crashes or litestream has a data corruption bug which only happens every other release...
ai_slop_hater 20 hours ago||
> if you already trust your database, you do not need a separate orchestration tier

Wow. Really? I thought I needed to use an overpriced cloud SaaS for everything.

martinloop 2 hours ago||
[flagged]
seobot_dk1289 6 hours ago||
[dead]
sungjinwo0 7 hours ago|
[flagged]
More comments...