Posted by sp_from_db 2 days ago
Amazing that the Postgres ecosystem got this software for “free” (as in at least a basic version of it is F/OSS, IIRC there wasn’t any core bits held back), and the extremely engineer-heavy company got to make money, AND they got bought out in true acquisition style by a larger player that truly benefits from the tech.
The Postgres ecosystem is pretty unique in its ability to produce a “boring” stable product, innovate, stay F/OSS, and create financial outcomes for participants.
Neon also only just disabled FPWs - so there is new substance here. We published a similar blog on Neon
If we compare the current state of the world to one in which they were acquired and then continued to put out more F/OSS, things look bad (which I assume is your implication). I choose to instead make the comparison to the world where we never see this tech and it stays proprietary. Sure, eventually someone in F/OSS might have gotten around to building this solution, but they pulled forward the future and we get to see and build on the result for free.
I don't follow: read requests are not served from the WAL. They read the current state of the page from the buffer cache, where the page is updated after the change (FPI or not) is written to the WAL.
In the past we relied on Postgres compute to periodically send a full page so reconstructive a page was always a bounded process. Once we turned it off (and got all those perf gains) we got another problem: unbounded page reconstruction which we had to solve separately.
Disaggregated storage and disaggregated compute have been an open trend in DBMS development for the last half-decade. This is an obvious move with modern computing paradigms, and the academic literature has a standard name for it.
This feels like "JAMStack" from Netlify happening all over again.
I tweeted about this in 2022, as a general trend, and also from the RocksDB meetup emphasizing disaggregated storage:
"Basic literacy" -> "Prompt Engineering"
"P2P networking" -> "Web3"
"Service-Oriented Architecture" -> "Microservices"
Maybe I'm old-man-yelling-at-cloud.
Since data is on s3 (or lake) you can perform direct to s3 type operations like data loading, reading this data by engines that are not Postgres and more
> in addition to disaggregated storage s3 is authoritative storage for older data
But if you say "Disaggregated Storage on S3" then you have the flexibility to change that to "Disaggregated Storage on FOOBAR" to avoid confusion.
I've never seen "lake" or adjacent terminology refer to S3 specifically like that vs other object storage. A data lake on Ceph would still be a data lake.
(My quibble would be that "lake" often refers to inconsistent or unstructured, and itself has always been a bit handwavy compared to "warehouse," whereas this is very structured data on object storage.)
Maybe I’m wrong, but AFAICT this is block (page) storage backed by S3, tuned for Postgres with some paxos-linked storage/caching servers sitting in front? Sounds good, but I’m not sure “lake” or “warehouse” is a word I’d choose… much closer to Litestream-with-reads, or the somewhat-famous “I ran out of RAM so I downloaded some more” blog article.
Many people just keep adding data and think "maybe it will be useful in future" till their system goes down.
Many of your data is essentially useless for anything in future.
You can simply have data retention policy and for most app this ensures your data does not grow top huge
I'm guessing with Neon, since their storage is a lakehouse, you get this for free.
Is the "without moving or duplicating" part actually a true statement? If the actual table state is only reconstructed by the pageserver, its not like Spark can just read it from S3.
However generally disaggregating storage makes HA simpler and allows for things like zero downtime patching: https://www.databricks.com/blog/zero-downtime-patching-lakeb...
Read replicas can be "shallow". You don't need to replicate all the data to create a replica. This allows to create them very very quickly (sub second).
All the extension still work. We don't support Citus today, but mostly because customers are not asking for it rather due to technical limitations. We support lots of extensions: https://docs.databricks.com/aws/en/oltp/projects/extensions
Operationally, how do you handle landing that large of a perf improvement? If my data store changed that much in a week it could break something.
After this change latencies are back to normal and throughput increased.
Great write up, cheers to the people involved.
This appears to only have any effect with datalake style installs, where storage is separate from compute.
Not going to have any effect on those small postgres installs for that generic one off app.
All of this to say that a ton of people are on some sort of managed cloud postgres where the compute is almost always separated from the storage even for the small instances.
Neon et al. will tell you they scale, and I am sure they can but the number of enterprises that actually exceed when can be put on a few large servers in pretty low. You gotta lock them in early so their orgs never develop the expertise to move off on the off chance they get big.
Small and large instances benefit from this performance optimization.