Stop syncing everything

Posted by neilk 3/31/2025

Stop syncing everything(sqlsync.dev)

656 points | 126 commentspage 2

smitty1e 4/2/2025|

"What are you syncing about?" https://youtu.be/0MUsVcYhERY?si=8gEMOo3nkd7SRX-Q

carlsverre 4/2/2025|

Finally got a chance to watch this. LMAO so good!

presentation 4/2/2025||

Another relevant project: https://zero.rocicorp.dev/

londons_explore 4/2/2025||

This approach has a problem when the mobile client is on the end of a very slow connection, yet the diff that needs to be synced is gigabytes, perhaps because it's an initial sync or the database got bloated for whatever reason.

A hybrid approach is to detect slow syncing (for example when sync hasn't completed after 5 seconds), and instead send queries directly to the server because there is a good chance the task the user wants to complete doesn't depend on the bloated records.

carlsverre 4/2/2025|

This is a great point. Solutions like Graft which involve syncing data to the edge work poorly when the dataset size is too large and not partitioned enough to be consistency partial.

This is why Graft isn't just focused on client sync. By expanding the focus to serverless functions and the edge, Graft is able to run the exact same workload on the exact same snapshot (which it can validate trivially due to its consistency guarantees) anywhere. This means that a client's workload can be trivially moved to the edge where there may be more resources, a better network connection, or existing cached state.

krick 4/2/2025||

Maybe it's just me, since people here in the comments apparently understand what this is, but even after skimming the comments, I don't. Some simplified API example would be useful either in the "marketing post" or (actually, and) in the github readme.

I mean, it's obviously about syncing stuff (despite the title), ok. It "simplifies the development", "shares data smoothly" and all the other nice things that everything else does (or claims to do). And I can use it to implement everything where replication of data might be useful (so, everything). Cool, but... sorry, what does it, exactly?

The biggest problem with syncing is, obviously, conflict resolution. Graft "doesn’t care about what’s inside those pages", so, obviously, it cannot solve conflicts. So if I'm using it in a note-taking app, as suggested, every unsynced change to a plain text file will result in a conflict. So, I suppose, it isn't what it's for at all, it's just a mechanism to handle replication between 2 SQLite files, when there are no conflicts between statements (so, what MySQL or Postgres do out of the box). Right? So, it will replace the standard SQLite driver in my app code to route all requests via some Graft-DB that will send my statements to external Graft instance as well as to my SQLite storage? Or what?

kubb 4/2/2025|

If they can’t successfully communicate the problem that they’re solving, chances are, they don’t know it themselves.

carlsverre 4/2/2025||

For a deep dive on Graft, a talk I did two weeks ago at Vancouver Systems just went live! You can watch it here: https://www.youtube.com/watch?v=P_OacKS2h5g

The talk contains a lot more details on how the transactional and replication layers of Graft work.

mrbluecoat 4/2/2025||

> Graft should be considered Alpha quality software. Thus, don't use it for production workloads yet.

Beta ETA?

carlsverre 4/2/2025|

ASAP :)

giancarlostoro 4/2/2025||

Reading their readme:

> Licensed under either of

> Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)

> MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)

> at your option.

I see this now and then, but it makes me wonder, why would I pick in this case Apache over MIT? Or is this software actually Apache licensed, but the developer is giving you greenlight to use it under the terms of the MIT? But at that point I don't get why not just license it all under MIT to begin with...

carlsverre 4/2/2025||

Per the Rust FAQ [1]:

> The Apache license includes important protection against patent aggression, but it is not compatible with the GPL, version 2. To avoid problems using Rust with GPL2, it is alternately MIT licensed.

The Rust API guidelines also recommend the same: https://rust-lang.github.io/api-guidelines/necessities.html#...

[1]: https://github.com/dtolnay/rust-faq#why-a-dual-mitasl2-licen...

From my perspective (as the author of Graft) my goal was to be open source and as compatible as possible with the Rust ecosystem. Hence the choice to dual license.

giancarlostoro 4/2/2025||

Thank you! That is helpful, and understandable from that context. I to try to follow the best standards surrounding the language I use for a given project.

CorrectHorseBat 4/2/2025||

One thing I can think of is that Apache gives you a patent grant and MIT doesn't

imagio 4/2/2025||

Very interesting! I've been hacking on a somewhat related idea. I'm prototyping a sync system based on pglite and the concept of replicating "intentions" rather than data. By that I mean replicating actions -- a tag and a set of arguments to a business logic function along with a hybrid logical clock and a set of forward & reverse patches describing data modified by the action.

As long as actions are immutable and any non-deterministic inputs are captured in the arguments they can be (re)executed in total clock order from a known common state in the client database to arrive at a consistent state regardless of when clients sync. The benefit of this I realized is that it works perfectly with authentication/authorization using postgres row level security. It's also efficient, letting clients sync the minimal amount of information and handle conflicts while still having full server authority over what clients can write.

There's a lot more detail involved in actually making it work. Triggers to capture row level patches and reverse patches in a transaction while executing an action. Client local rollback mechanism to resolve conflicts by rolling back local db state and replaying actions in total causal order. State patch actions that reconcile the differences between expected and actual outcomes of replaying actions (for example due to private data and conditionals). And so on.

The big benefits of this technique is that it isn't just merging data, it's actually executing business logic to move state forward. That means it captures user intentions where a system based purely on merging data cannot. Traditional crdt that merges data will end up at a consistent state but can provide zero guarantees about the semantic validity of that state to the end user. By replaying business logic functions I'm seeking to guarantee that the state is not only consistent but maximally preserves the intentions of the user when reconciling interleaved writes.

This is still a WIP and I don't have anything useful to share yet but I think the core of the idea is sound. Exciting to see so much innovation in the space of data sync! It's a tough problem and no solution (yet) handles the use cases of many different types of apps.

carlsverre 4/2/2025|

Glad you enjoyed Graft! The system your describing sounds very cool! It's actually quite similar to SQLSync. The best description of how SQLSync represents and replays intentions (SQLSync calls them mutations) is this talk I did at WasmCon 2023: https://www.youtube.com/watch?v=oLYda9jmNpk

imagio 4/2/2025||

Cool! That's an interesting approach putting the actions in wasm. I'm going for something more tightly integrated into an application rather than entirely in the database layer.

The actions in my prototype are just TS functions (actually Effects https://effect.website/ but same idea) that can arbitrarily read and write to the client local database. This does put some restrictions on the app -- it has to define all mutations inside of actions and capture any non-deterministic things other than database access (random number, time, network calls, etc) as part of the arguments. Beyond that what an app does inside of the actions can be entirely arbitrary.

I think that hits the sweet spot between flexibility, simplicity, and consistency. The action functions can always handle divergence in whatever way makes sense for the application. Clients will always converge to the same semantically valid state because state is always advanced by business logic, not patches.

Patches are recorded but only for application to the server's database state and for checking divergence from expected results when replaying incoming actions on a client. It should create very little load on the backend server because it does not need to execute action functions, it can just apply patches with the confidence that the clients have resolved any conflicts in a way that makes the most sense.

It's fun and interesting stuff to work on! I'll have to take a closer look at SQLSync for some inspiration.

carlsverre 4/2/2025||

Woah that's awesome. Using Effect to represent mutations is brilliant!

Any chance your project is public? I'd love to dig into the details. Alternatively would you be willing to nerd out on this over a call sometime? You can reach me at hello [at] orbitinghail [dotdev]

imagio 4/2/2025|||

I haven't put it up in a github repo yet, it's not finished enough, but I will after spending a little more time hacking on it. I'll try to remember to come back here to comment when I do =)

Using effect really doesn't introduce anything special -- plain async typescript functions would work fine too. My approach is certainly a lot less sophisticated than yours with Graft or SQLSync! I just like using Effect. It makes working with typescript a lot more pleasant.

imagio 4/3/2025|||

As promised, here's the code. I decided to call it Synchrotron. Not ready for use yet but there is design/planning in the readme and a suite of passing tests showing that the approach can work. https://github.com/evelant/synchrotron

carlsverre 4/3/2025||

Sweet!! Will check it out shortly. Thanks for sharing :)

mhahn 4/2/2025||

I looked at using turso embedded replicas for a realtime collaboration project and one downside was that each sync operation was fairly expensive. The minimum payload size is 4KB IIRC because it needs to sync the sqlite frame. Then they charge based on the number of sync operations so it wasn't a good fit for this particular use case.

I'm curious if the graft solution helps with this. The idea of just being able to ship a sqlite db to a mobile client that you can also mutate from a server is really powerful. I ended up basically building my own syncing engine to sync changes between clients and servers.

carlsverre 4/2/2025|

For now, Graft suffers from the same minimum payload size of 4KB. However, there are ways to eliminate that. I've filed an issue to resolve this in Graft (https://github.com/orbitinghail/graft/issues/35), thanks for the reminder!

As for the more general question though, by shipping pages you will often ship more data than the equivalent logical replication approach. This is a tradeoff you make for a much simpler approach to strong consistency on top of arbitrary data models.

I'd love to learn more about the approach you took with your sync engine! It's so fun how much energy is in the replication space right now!

theSherwood 4/2/2025|

This is a very interesting approach. Using pages as the basic sync unit seems to simplify a lot. It also makes the sync of arbitrary bytes possible. But it does seem that if your sync is this coarse-grained that there would be lots of conflicting writes in applications with a lot of concurrent users (even if they are updating semantically unrelated data). Seems like OT or CRDT would be better in such a use-case. I'd be interested to see some real-world benchmarks to see how contention scales with the number of users.

carlsverre 4/2/2025|

Thank you! You're 1000% correct, Graft's approach is not a good fit for high write contention on a single Volume. Graft instead is designed for architectures that can either partition writes[^1] or can represent writes as bundled mutations and apply them in a single location that can enforce order (the SQLSync model).

Basically, Graft is not the full story, but as you point out - because it's so simple it's easy to build different solutions on top of it.

[^1]: either across Volumes or across pages, Graft can handle automatically merging non-overlapping page sets

More comments...