Top
Best
New

Posted by jorangreef 2 days ago

Spiral(spiraldb.com)
258 points | 87 commentspage 2
dwroberts 2 days ago|
> Remember that uncanny valley between 1KB and 25MB? The problem isn't the sizes—it's that Second Age systems force you to choose between two bad options: inline the data (killing performance) or store pointers (breaking governance). Spiral eliminates this false choice. We store 10KB embeddings directly in Vortex for microsecond access, intelligently batch 10MB blocks of images for optimal S3 throughput, and externalize 4GB videos without copying a single byte. One system, no compromises.

No compromises but isn’t ‘externalising’ a large video the equivalent of storing a pointer in the first example? Can’t really see any other way to understand what that means (it goes to an external system and you store where it is)

rubenvanwyk 2 days ago||
Interesting that Joran from Tigerbeetle posted this? So must be legit.
jorangreef 2 days ago|
I’ve known Will for some time while they’ve been in stealth.
derekhecksher 2 days ago||
The AnyBlox paper from the folks at TUM, and linked to in the post, is a bit more interesting, imo, since it looks to solve the data systems x storage format problem in composable data architectures - https://gienieczko.com/anyblox-paper
aduffy 2 days ago|
The AnyBlox paper has some very cool ideas, and the authors are friends.

In the paper you'll notice a large portion of it analyzes Vortex, both standalone and embedded. Definitely worth a read.

all2 2 days ago||
Spelling error "sttill"

> P.S. If you're sttill managing data in spreadsheets, this post isn't for you. Yet.

---

Since I discovered the ECS pattern, I've been curious about backing it with a database. One of the big issues seems to be IO on the database side. I wonder if Spiral might solve this issue.

harwoodr 2 days ago||
Have a look at something like spacetimeDB - caveat, I've only read about it and not directly used it:

https://github.com/ClockworkLabs/SpacetimeDB

all2 2 days ago||
This was part of my inspiration. They do some interesting things with their systems, like hot reloads, that makes the dev loop a lot tighter.
lordnacho 2 days ago||
If the ECS data is grid-like, perhaps you could use a columnar database for time series?

Then you could save every single state change and scroll back and forth. But I'm not sure if you were looking for that.

all2 2 days ago||
This would make netcode state rollback easy. But I'm not sure the DB is the right place for that.
4ndrewl 2 days ago||
The three eras of database systems starts with a client-server Postgres, but missed the daddy of the generation before that - xBase (ie dBase, FoxPro etc).
khaledh 2 days ago|
It goes way before that. It starts with IDS (Integrated Data Store) from GE (1964), which was a network database system. Next was IBM's hierarchical database system IMS (Information Management System, 1966), still in use today. Then the CODASYL model (late 1960s), which was an effort to standardize the network model. And then Codd came up with the relational model in the early 70s, upon which an explosion of database systems were built (first is IBM System R, SQL, Oracle, DB2, Ingres). Then came the PC-based database systems you mentioned.
4ndrewl 2 days ago||
Oh for sure. To suggest we're only on generation 3 of "databases" is way off the mark.
redwood 2 days ago||
So it's for low change rate data that needs to be bulk processed during ML model training. Cool. But hardly the same thing as what you need for powering live AI applications... which is what I assumed this was upon reading the intro and the mention of Postgres..

Postgres (and MongoDB) are the king and prince of data due to their transactional capabilities.

dwb 2 days ago||
If you don't clearly detail what your new tech product or system is bad at, as well as what it's good at, I'm not interested. So much of engineering is about navigating the inevitable tradeoffs. Marketing should have no place in engineering.
rubenvanwyk 2 days ago||
How does Vortex compare to Lance? I imagine Lance is already a good solution for AI on CPUs.
More comments...