Tailscale Traces Database Corruption to 16y/o SQLite WAL-Reset Bug

Posted by ropbear 4 hours ago

Tailscale Traces Database Corruption to 16y/o SQLite WAL-Reset Bug(tailscale.com)

430 points | 68 commentspage 3

riknos314 3 hours ago|

> Whenever corruption occurred, we had to stop the control plane process on the shard while we repaired or restored the database. This was painful for tailnets on that shard, because their entire control plane disappeared during that recovery window.

Gotta love single points of failure...

tptacek 1 hour ago||

This is maybe one of the purest examples of the Bell Curve Meme in software engineering. The things you would do to the system Tailscale operates to eliminate all single points of failure (generally, and in the specific case where, where the "single point of failure" applies only to a small cohort of customers) would make the system less resilient, and increase failures.

Generally, you do complex distributed systems without on-paper single-points-of-failure anywhere when you absolutely have to, because those systems don't have transient failures. That's not mesh networks like Tailscale at all.

As always: https://how.complexsystems.fail/

arjie 2 hours ago|||

You don’t need the control plane most of the time. I had a zero downtime headscale upgrade because once the nodes negotiate through the control plane they can talk to each other all the time. The data plane is peer to peer.

It’s problematic because you can’t run connections but it doesn’t stop the world.

kccqzy 3 hours ago|||

What are some solutions to avoid database corruption being single points of failure? I can’t think of any off the top of my head. I don’t think people typically consider database corruption to be a kind of failure common enough to design for, unless you have unusual requirements.

AlotOfReading 2 hours ago||

The general answer to this is Byzantine consensus, which cryptocurrency blockchains are designed to solve. If your nodes are willing to fail a little more politely (e.g. no lying, immediately crashing, etc) you can use something cheaper like raft/paxos.

But yeah, it's a lot cheaper to build a reliable system than it is to be resilient.

spockz 2 hours ago||

The shard was already a way to make it not a single point of failure.

Spivak 31 minutes ago||

This is a great example of outages looking different from the perspective of the operator vs the user. Because there's many shards the blast radius of failure is contained to a small subset of users but for those users it's an outage. The way it's designed you can't lose any shards without impacting users. Compare to say Elasticsearch where it's possible to lose nodes and lose shards without the user noticing. One approach isn't universally better than the other.

dzonga 2 hours ago||

you gotta admire the power of using json/b and simple KV stores.

so many people sleep on that.

pstuart 3 hours ago||

I imagine the SQLite eschews AI generated code, but using it for testing (vulnerability, performance, etc) would seem like an easy win.

I know their proprietary testing framework is their secret sauce so we may never know...

d-us-vb 2 hours ago|

Richard Hipp's recent talk at Software Should Work explains that AI agents have been testing SQLite and they've gotten a deluge of new bug reports from the fuzz-like testing they can do. But they do not do this in house; hobbyists and other organizations do this in their own internal agent-driven fuzzing.

ec109685 3 hours ago|

While technically true as written, it seems to downplay the significance:

> The bug is a data race with tight timing constraints. It is unlikely to occur in common use.

A large customer did experience this corruption, so it's important for people with tailscale's setup update immediately.

> The developers have never been able to reproduce the bug organically and had to add special testing logic to SQLite that deliberately triggers the circumstances of the the bug in order to verify that the issue has been fixed.

Ariarule 3 hours ago||

Odd not to highlight the sentence where they answer the obvious question "Why Tailscale in particular?":

> They also explained why we were more likely to hit the bug than other SQLite users: we take manual control of the checkpointing process, and we checkpoint very aggressively. Even a bug triggered by a rare condition was bound to hit us eventually.

dboreham 2 hours ago||

Quick note that data corruption bugs that are impossible to reproduce are not uncommon (perhaps they're the norm). So some amount of head scratching trying to figure out a plausible scenario by which the system could get into the state represented by the smoking remains is often required. Then you attempt to force it into the supposed bad state by modifying code paths accordingly. So the approach used in this case is clever, but it's not particularly unusual in the world of data stores.