There's a whole thing this days about building solvers (e.g. SAT or Ising) out of exotic hardware that does compute in memory. A while back I wondered if one could leverage distributed DB logic to build solvers for massive problems, something like compute in DB.

I love reading posts like these.

I will still reach for a database 99% or the time, because I like things like SQL and transactions. However, I've recently been working on a 100% personal project to manage some private data; extracting insights, graphing trends, etc. It's not high volume data, so I decided to use just the file system, with data backed at yaml files, with some simple indexing, and I haven't run into any performance issues yet. I probably never will at my scale and volume.

In this particular case having something that was human readable, and more importantly diffable, was more valuable to me than outright performance.

Having said that, I will still gladly reach for a database with a query language and all the guarantees that comes with 99% of the time.

Very interesting, I'd never heard of JSONL before: https://jsonlines.org/

Also notable mention for JSON5 which supports comments!: https://json5.org/

In many cases not. E.g. for caching with python, diskcache is a good choice. For small amounts of data, a JSON file does the job (you pointed to JSONL as an option). But for larger collections, that should be searchable/processable, postgres is a good choice.

Memory of course, as you wrote, also seems reasonable in many cases.

> The index format is simple: one line per record, exactly 58 bytes: <36-char UUID>:<20-digit byte offset in data file>\n.

It would be much better to write all of this as binary data, omitting separators.

• Since it’s fixed-width and simple, inspecting the data is still pretty easy—there are tools for working with binary data of declared schema, or you could write a few-liner to convert it yourself. You don’t lose much by departing ASCII.

• You might want to complicate it a little by writing a version tag at the start of the file or outside it so you can change the format more easily (e.g. if you ever add a third column). I will admit the explicit separators do make that easier. You can also leave that for later, it probably won’t hurt.

• UUID: 36 bytes → 16 bytes.

• Offset: 20 bytes (zero-padded base-ten integer) → 8 bytes.

• It removes one type of error altogether: now all bit patterns are syntactically valid.

• It’ll use less disk space, be cheaper to read, be cheaper to write, and probably take less code.

I also want to register alarm at the sample code given for func FindUserBinarySearch. To begin with, despite a return type of (*User, error), it always returns nil error—it swallows all I/O errors and ignores JSON decode errors. Then:

  entryID := strings.TrimRight(string(buf[:36]), " ")

That strings.TrimRight will only do anything if your data is corrupted.

  cmp := strings.Compare(entryID, id)

Not important when you control the writing, but worth noting that UUID string comparison is case-insensitive.

  offsetStr := strings.TrimLeft(string(buf[37:57]), "0")

Superfluous. ParseInt doesn’t mind leading zeroes, and it’ll probably skip them faster than a separate TrimLeft call.

  dataOffset, _ := strconv.ParseInt(offsetStr, 10, 64)

That’s begging to make data corruption difficult to debug. Most corruption will now become dataOffset 0. Congratulations! You are now root.

I agree that using a database in many cases is overkill. And I don't understand at all who and why uses so-called in-memory databases.

While this is certainly cool to see. And I love seeing how fast webservers can go.. The counter question "Do you even need 25,000 RPS and sub-ms latency?" comes to mind.

I don't choose a DB over a flat file for its speed. I choose a DB for the consistent interface and redundancy.

I need a filesystem that does some database things. We got teased with that with WinFS and Beos's BFS, but it seems the football always gets yanked away, and the mainstream of filesystems always reverts back to the APIs established in the 1980s.

I'm a big fan of using S3 as a database. A lot of apps can get a lot of mileage just doing that for a good chunk of their data; that which just needs lookup by a single field (usually ID, but doesn't have to be).

In order to ask this question it's important to understand the lifecycle of the data in question. If it is constantly being updated and requires "liveness" (updates are reflected in queries immediately), the simple answer is: yes, you need a database.

But if you have data that is static or effectively static (data that is updated occasionally or batched), then serving via custom file handling can have its place.

If the records are fixed width and sorted on the key value, then it becomes trivial to do a binary search on the mmapped file. It's about as lightweight as could be asked for.