Posted by smiths1999 12 hours ago
I'm currently building a personal knowledge graph server a mix of Notion's custom entities via JSON schema and Obsidian markdown+backlinked references. It's working well, but I suspect your product might be a better fit.
I do have one question regarding permissions: how would you recommend modeling a hierarchical access system in a graph database? Specifically, if a user is granted access to a document, they should automatically have access to all its child documents within that workspace. Is there a standard way to model this 'subtree' permission logic, or perhaps a more efficient approach you'd suggest?
Really impressed with the product good luck with it!
Anyways, thanks for checking it out! Really appreciate it. Good luck with your project!
zig build sqlite-benchmark
Medium (100K nodes)
+----------------------+-----------+---------+---------+
| Workload | LatticeDB | SQLite | Speedup |
+----------------------+-----------+---------+---------+
| 1-hop traversal | 5.7μs | 16.1μs | 2.8x |
| 2-hop traversal | 30.1μs | 59.4μs | 2.0x |
| 3-hop traversal | 171.1μs | 228.8μs | 1.3x |
| Variable path (1..5) | 82.2μs | 5.8ms | 70.4x |
+----------------------+-----------+---------+---------+
Very different from the comparison on github and the website.Given that on-disk data structures are similar to SQLite, I expect the competition from other "graph on sqlite" projects when they co-opt the techniques in LatticeDB.
My team and I built it over the years, and it's open source (AGPL). Here is how it works: https://community.qbix.com/t/qbix-streams-as-a-graph-databas...
Our database abstraction layer (and optional ORM) has been battle tested in production for millions of users, and has 3 adapters: Sqlite, Postgres, MySQL/MariaDB. And recently it even added vector search for ranking results by similarity: https://github.com/Qbix/Platform/tree/main/platform/classes/...
Documentation for the database layer is here: https://qbix.com/platform/guide/database
PS: If you do use a relational database for storing graph data, you're going to have a lot of duplication in some public keys. I highly recommend putting ZFS underneath, to help with deduplication. ZFS uses zstd, developed at facebook, and also can encrypt your data at rest (don't use the relational database to do the encryption, otherwise deduplication doesn't work).
In general though, the goal for this was single writer multiple readers. That was a design decision to keep things simple.
As for the tool, it scratches an itch I've been having, I'll give it a go soon.
The initial phases of building I would build out piece by piece. For example, building out the file system interactions I would have claude build a feature and explain how it worked in an educational manner (e.g., like it was a section in a book on latticedb internals). I would then read through the code. This was a great way to learn and build, simultaneously.
In the later stages, where the features and work was more complex, I would spend more time discussing, instructing, and verifying, but less time understanding the actual implementation. I'll give you an example. It's been a long time since I've handwritten SIMD code. I could try and review claudes output, but I am certain I'd miss any subtle bugs that may exist. I found it more productive to assume the code was right and focus on thinking about how I would verify that. Benchmarking, playing with latticedb, etc. were my primary tools for verifying the work. I could run a benchmark and see performance was great. Then I'd explore the test vectors and realize they were trivial, completely invalidating the benchmark results. So we would go back to the drawing board, create a new benchmark set, see results weren't great, and evaluate what was wrong with the implementation. Sometimes features would take days to get out just because of the iteration loop.
One of the motivating use cases for me was experimenting with agentic memory. I use latticedb as the backing data store. Finding related memories is traversing the graph (kind of like graph RAG).