Top
Best
New

Posted by emschwartz 3 days ago

Litestream VFS(fly.io)
371 points | 83 commentspage 2
JSR_FDED 3 days ago|
From a Litestream user’s perspective:

Litestream continues to work as always, making continuous backups to S3.

Like always, I can restore from those backups to my local system.

But now I have the option of doing “virtual restores” where I can query a database backup directly on S3.

jauntywundrkind 3 days ago||
So much fun streaming/sync/cdc stuff happening, all so cool. Having an underlying FUSE driver doing the Change Data Capture is really neat. This looks like such an incredibly lightweight way to remote-connect to sqlite. And to add a sort of exterior transaction management.

Different use case, but makes me think of sqlite Rewrite-it-it-Rust Turso announcing AgentFS. Here the roles are flipped, sqlite is acting as a file store to back FUSE, to allow watching/transaction-managing the filesystem/what agents are doing. Turso also has a sick CDC system built in, that just writes all changes to a cdc table. Which is related to this whole meta question, of what is happening to my sqlite DB. https://turso.tech/blog/agentfs

tptacek 3 days ago|
Just to be clear, the underlying FUSE thing is LiteFS, not Litestream; nothing described in this post needs a FUSE filesystem, just a small SQLite VFS plugin library.
jauntywundrkind 3 days ago||
Thanks Thomas.

To just drop the relevant paragraph that addresses my un-clarity/in-correctness (and which is super fun to read):

> Litestream v0.5 integrates LTX, our SQLite data-shipping file format. Where earlier Litestream blindly shipped whole raw SQLite pages to and from object storage, LTX ships ordered sets of pages. We built LTX for LiteFS, which uses a FUSE filesystem to do transaction-aware replication for unmodified applications, but we’ve spent this year figuring out ways to use LTX in Litestream, without all that FUSE drama.

tptacek 3 days ago||
You got it. Yeah, Ben built LiteFS a year or two ago, which is why he designed LTX. But using LiteFS required people to set up FUSE filesystems, which was too complicated for a lot of people, so Litestream got way more uptake. This past year he's been harvesting all the good stuff from LiteFS that doesn't require FUSE and building it into Litestream.

The easiest way so far to understand the split between Litestream and LiteFS: Litestream is an operational tool, for backup and restore. LiteFS is a method for doing online leader/follower replica clusters.

pkhuong 3 days ago||
I remember when Litestream not being a VFS was a plus https://news.ycombinator.com/item?id=29461406 ;)
tptacek 3 days ago|
It still is! I don't know how much clearer we could be about this point. :)

Litestream does not require a VFS to work. It still does all the cool stuff it did before; in fact, it does those things much better now, even without the VFS.

dangoodmanUT 3 days ago||
I'm glad they did this! I've always thought VFS was a better fit for the objectives of Litestream than the original design.

SQLite VFS is really cool tech, and pretty easy to work with (IMO easier than FUSE).

I had made a _somewhat similar_ VFS [1] (with a totally different set of guarantees), and it felt pretty magical how it "just worked" with normal SQLite

[1] https://github.com/danthegoodman1/gRPSQLite

orliesaurus 3 days ago||
Been tinkering with litestream... the read-only VFS is neat but I'm curious about eventual write capabilities... using VFS for distributed DBs could unlock some interesting patterns.

ALSO I'm thinking about mixing this with object store caching... maybe combining memfs with remote metadata; would love to see more details on performance.

BUT I might be overthinking it... just excited to see SQLite exploring beyond local files...

reactordev 3 days ago|
Opens up a whole new can of worms. Transactions come to mind. Who would be responsible for coordinating? If two nodes wrote to the table conflicting information at the same time, who wins?
benbjohnson 3 days ago||
Author here. We've done some proof-of-concept work on creating distributed leases using S3. We have some use cases internally where we've considered adding write capabilities to the VFS but we haven't started any work on it yet.
reactordev 3 days ago|||
It’s a tricky problem that goes beyond the fs as you know. Since it’s cloud and since it’s distributed, a manager wouldn’t be that far fetched that could issue CRDT like messages across the cluster of nodes to issue a “write” or utilize the sync mechanism you have to propagate a “master” db that you write to (aggregate or designate, either way). I did some work on this on a go based database graph and ended up doing a gossip sync crdt message bus.
withinboredom 2 days ago||
If latency isn't important. Something like this could work. For applications where latency or linearizability is important, it probably won't work.
hobo_mark 3 days ago|||
Is this VFS for read-only databases? Or can I query a database that has a single litestream writer somewhere continously making updates and backing them up to S3?
benbjohnson 3 days ago||
The VFS is read only but it will continuously poll for new updates so if you have a writer somewhere else using regular Litestream then it will pick up those updates automatically.
seigel 2 days ago||
Just a heads up for people. This will not work (as far as I could get working) with a system running 0.3.x backups to s3 and trying to restore from them.
joshstrange 3 days ago||
I work with many distributed, often offline, hosts with varied levels of internet speeds. Does this do any offline caching? Like if I load a vfs litestream database on one of my nodes and it goes offline can it still query or will it fall over unless the data was recently fetched?
benbjohnson 3 days ago|
Author here. It will cache pages in memory right now but it doesn't do disk-based caching currently. That's a good idea though and probably not too difficult to implement. Feel free to post a feature request as a GitHub issue on the project.
honktime 3 days ago||
I dont fully understand this, would this be useful for scaling sqlite on systems that have really high read needs and a single writer? I thought that was what LiteFS was for, or am i off on that too?
tptacek 3 days ago||
No, you're right: scaling "out" SQLite is what LiteFS is about, and this is about (significantly) improving operational capabilities for single-server (or multi-independent-server, like distributed cache) SQLite deployments.
davnicwil 3 days ago||
I have to say I'm similarly not really sure I'm getting it, and I've been following litestream pretty closely over the years (not an expert but certainly familiar).

I think what we're getting here is a way to just spin up a local shell / app and run arbitrary queries from any point in time over the network without having to sync the full prod database. I guess with LiteFS you would have to do this, or pre-plan to do this, it's not totally on-demand.

Or said another way, do things locally as though in prod without having to ssh to prod and do it there (if you even can, I guess if 'prod' is just s3 you can't really do this anyway so it's an entirely new capability).

@benbjohnson is this right? I humbly suggest adding a tl;dr of the main takeaway up top of the post to clarify. Love your work on litestream, thanks for what you do!

tptacek 3 days ago||
Your second paragraph is correct, with the added proviso that you can build features in your own application tooling to take advantage of the same capability.

One reason you're not getting such a clear usage statement at the top of this post is, it's an early feature for a general-purpose capability. I think we might rather get other people's takes about what it's most useful for? There are some obvious use cases, like the one you just identified.

davnicwil 3 days ago||
very fair! Thanks for confirming
itissid 3 days ago||
Really nice. We should have this as an add-on to https://app.codecrafters.io/courses/sqlite/overview It can probably teach one a lot about the value of good replication and data formats.

If you are not familiar with data systems, havea read DDIA(Designing Data Intensive Applications) Chapter 3. Especially the part on building a database from the ground up — It almost starts with sthing like "Whats the simplest key value store?": `echo`(O(1) write to end of file, super fast) and `grep`(O(n) read, slow) — and then build up all the way to LSMTrees and BTrees. It will all make a lot more sense why this preserves so many of those ideas.

hintoftime 3 days ago|
Does this mean that I can run an application in K8s via one or many horizontally scaled pods all running off DB in s3? No StatefulSet required?
dpedu 3 days ago||
I was doing something similar just the other day and came across sqlite-s3vfs[0]. It is likewise a SQLite VFS that translates IO to S3 api calls. However, that project is only for python and seemingly abandoned. Additionally, if you want more than one writer, you'd have to coordinate that yourself, afaik.

[0]: https://pypi.org/project/sqlite-s3vfs/

benbjohnson 3 days ago||
Author here. The VFS support right now is currently read only so it's useful for something more like a shared cache of data.
More comments...