Posted by smiths1999 13 hours ago
What are some of the scales of the data you've been able to test this design on so far?
What was the most interesting part of designing it for you?
Most interesting part is a tough one. From a learning perspective the beginning was incredibly interesting because I was spending a lot of time learning about how other DBs work. Even something as relatively simple as writing to disk had a lot more complexity to it than I initially anticipated.
I used LLMs extensively in building this, and the other interesting part was seeing how they failed. I've always been a proponent that tests are no guarantee of quality code, and working with LLMs has only reinforced it. They often write superficial tests. Sometimes a suite of tests would pass, but when I would actually play around with the feature it was clearly broken. LLMs certainly enabled me to build something of this scope, but it was far from "build a graph DB and notify me when you are done"
Couple of corrections:
* LadybugDB has revamped the Kuzu WAL design. It shouldn't be hard to build WAL based replication
* 19ms vs 39us - like the author says these are vastly different systems and the benchmark methodology may not be comparable.
We've mostly focused on query plan optimizations, not so much the micro query operator optimizations.
The 0.20.x end of the month release should have some interesting optimizations.
* Prepared statements will cache query plans and result vectors. So you don't pay malloc costs
* SIMD optimizations for filter. More to come in the next release.For those into the `pip install ...` flow and kuzu, is gfql: we started around the same time in a non-VC-funded oss manner with overlap in key architectural ideas:
- cpu columnar vectorized engine + optionally the only open source gpu engine mode for bigger graphs / faster queries
- removes the need for a database / file: pure compute-tier engine you can write to parquet/json if you want, plays with parallel reader/writers in simple ways b/c that, and TBD iceberg
- adds full graph analytic pipeline support, eg, for feature engineering in real-time fraud & memory pipelines
- also millisecond/submillisecond times on small graphs like that small 100K edge graph benchmark
Main box not formally checked is streaming. Funny enough, we're designed for GPU firehose workloads, so would be fun to demo and see what gaps are left.
However, I really miss the content posted by the KuzuDB team on their YouTube channel.