Top
Best
New

Posted by poly2it 18 hours ago

Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD(malisper.me)
264 points | 124 commentspage 2
patkepa 8 hours ago|
Question, does having it in pure rust, opens possibility of embedding pgrust directly into binary, making it an alternative to SQLite/turso?
malisper 6 hours ago||
It absolutely can be embedded. The bigger enabler is replacing the process-per-connection model with a thread-per-connection model. Projects like pglite[0] had to give up concurrency because of it. We also support compiling to wasm so you can embed it in the browser too, which is what powers pgrust.com

[0] https://github.com/electric-sql/pglite

[1] https://pgrust.com/

wkoszek 4 hours ago|||
I'd help with testing if you helped them add this. I'd be interested in this.
ComputerGuru 2 hours ago||
Look into tursodb for this.
luciana1u 8 hours ago||
300x faster is nice, but I mostly clicked to see if operator fusion finally explains why my GROUP BY still feels like it is doing the work by hand.
Lucasoato 12 hours ago||
I’m curious to see how this compares to pgColumnar or other OLAP extensions.
malisper 12 hours ago|
At least in terms of speed, we're much faster on clickbench: https://benchmark.clickhouse.com/#system=+_b|pnc|pgrs|gQ|saB...
wiradikusuma 11 hours ago||
I think in addition to making it faster, it would be useful if it could be made "leaner," e.g. can run better on lower-spec hardware than PG.
claytonjy 10 hours ago|
does one not imply the other? if it can run faster in the same hardware, it should also run as fast on lower spec hardware
smolder 10 hours ago|||
No, it doesn't. Different algorithms can vary pretty wildly in performance based on the design of the hardware they run on. For instance cache sizes can make one implementation of a sort on a certain sized dataset faster or slower than another. You can have a theoretically fast algorithm that just isn't as cache efficient as a theoretically slower one in big-O terms. All levels of the memory hierarchy as well as storage have specific bandwidths and latencies that inform the real world performance results. Parallelism is another issue. Many cores can do work very fast when you're careful about how you split up work between them, taking into consideration the synchronization latency and individual cache sizes and so on. The best approach for doing work on 64 cores can be dramatically different from what works best on 1 or even 2.
Tuna-Fish 10 hours ago||||
That is not a given. A database server can run faster on better hardware because it more effectively and aggressively caches things in memory, which can hurt it on lower-spec systems. Or it can better utilize SIMD instructions that are not present on the low end. Or it is more effective at utilizing more threads, but is slower when run at a low threadcount, etc etc etc.
wiradikusuma 10 hours ago|||
If I'm not mistaken ClickHouse's min spec is quite steep.
KolmogorovComp 11 hours ago||
Thanks to the authors for choosing a license that respect users freedom, on top of being an awesome technical project.
jjice 10 hours ago||
While I do like pgrust's license, I do feel like it's kind of wrong to port in such a direct way and change the license. I guess this isn't a fork, but it kind of is? It looks like according to this post [0], they did a Claude Fable + Opus re-write. I know that legally this is seemingly a valid way to do things and avoid copyright, but it feels wrong to me. I don't even necessarily think that my feelings are correct, but standing on the backs of giants and using an LLM to "reimplement" the code (not a clean room implementation) feels like it's it _shouldn't_ be a valid way to avoid copyright or allow for license changes.

Now, I don't know how MIT -> AGPL re-licensing specifics work, but still.

[0] https://malisper.me/postgres-in-rust-regression-suite/

nz 7 hours ago||
My understanding is that they used c2rust, and then told the LLM/Agents to make the code more idiomatic rust, while also using the PG test suite as a feedback mechanism. This is almost certainly a derived work (and thus a fork, and should thus have the original license and copyright preserved).

For example, if I compiled PG into x86-64 assembly, and then decompiled it into C (via, say, IDA), and then polished that decompiled C code into very readable C code, it is still a derived work. For some reason, people think that if you include an LLM or Agent, copyright can be ignored, and plagiarism is now no longer possible.

It is similar to the crypto-folks thinking that if you use crypto, you no longer have to pay taxes, because the internet/computers make all inconvenient realities go away.

Honestly, such flagrant and arrogant copyright violations make it hard for me to take the project seriously, because it seems like a desperate stunt for attention (which itself may be a solid business move, but that is besides the point).

Put differently, if one were to fork pgrust, strip away the new license and copyright, and restore the original PG license and copyright (while also adding malisper+team to that copyright), they would face no legal consequences at all. In fact, they would probably be a less legal risk than the pgrust team.

malisper 6 hours ago||
> My understanding is that they used c2rust, and then told the LLM/Agents to make the code more idiomatic rust, while also using the PG test suite as a feedback mechanism.

This is correct

> and thus a fork, and should thus have the original license and copyright preserved)

This is not correct. The Postgres license is permissive. We need to include a copy of the license (which we do in the NOTICE file[0]) but we CAN relicense the Postgres code however we want as long as we meet the requirements of the license. pgrust is a derived work of Postgres, but Postgres allows derived works to be under a different license.

[0] https://github.com/malisper/pgrust/blob/main/NOTICE

imhoguy 9 hours ago||
Does that license protect anything legally? This is written by AI under some initial guidance. Prompts may be protected but the output of unlicensed knowledge model corpus?
up2isomorphism 5 hours ago||
300x if it is true you will be just busy dealing with you customers rather than pitching here. Also since it is a vibe coded project, if you are really that good , you should even need to related yourself with Postgres. Who will want related itself to something that is 300x slower than itself?
wkoszek 4 hours ago|
You use stability of PostgreSQL to open a pitch. Also most people use PostgreSQL because someone out there was a fan, proponent and champion of Postgres. Otherwise they'd be on MySQL or Oracle.

And your sales folks would call and say: "No need to change anything, we still run PostgreSQL, and ours is just called pgrust, but it's N times as fast".

jiggawatts 6 hours ago||
Batch mode execution has been in Microsoft SQL Server for a while and just recently gained AVX-512 support.

I’ve done some experiments replacing spatial SQL queries with custom vectorised batch mode code in C# and the speed up was astonishing.

The people dubious about these claims have no idea what their computers are really capable of.

Seattle3503 9 hours ago||
pgrust looks interesting. Could it be used as a library by someone who wants a new DB for each integration test in their Rust test suite?
malisper 8 hours ago|
One of the new features we recently built is "test mode". This brings cloning a template db from 100ms down to <10ms making it much better for tests.

If you're interested in trying it out, please reach out to me at malis@pgrust.com

cognitiveinline 15 hours ago||
pgrust seems to have good momentum. AGPL is an odd license for a non web project. Postgres is MIT-like, and that drove it's adoption.

Have pgrust folks reconsidered this? Else, IMO we can have an independant rust port of pgrust, which can be MIT, which will garner more attention.

malisper 13 hours ago||
Author here. At least for databases, AGPL (or stricter) has become standard. The issue is it's so easy for megacorps (Amazon, Google, etc) to take a permissively licensed product and monetize it at the expense of the original standard.

For instance, Mongo, Cockroach, and Materialize have all gone source available. We picked AGPL because it's the best balance between open source and prevents Amazon from just repackaging it and selling it.

If AGPL is an issue for anyone, we would be happy to dual-license under a commercial license.

mey 13 hours ago|||
There is nothing wrong with wanting to be compensated for your work, but for people like myself which use a cloud managed DB solution (GCP CloudSQL PostgreSQL) it means something like this would never be available.

I consider AGPL a poison pill in my work. That is not true with a suitable commercial license, although I expect a lot more commercial product (support/features/etc). As you note, your objective is to prevent commercialization of your software, but radically speeding up analytics is primarily a concern of large organizations so it seems like a mismatch in purpose.

JamesSwift 10 hours ago|||
The whole point is to force the cloud providers to go through a contract negotiation for licensing rather than keep the door open for them. Nothing prevents the parent project from specially licensing to individual parties.
hobofan 11 hours ago||||
> which use a cloud managed DB solution (GCP CloudSQL PostgreSQL) it means something like this would never be available

That's up to GCP then. I'm sure OP would be willing to license pgrust to them for an appropriate price.

shawnz 9 hours ago||
What's the incentive for third parties to contribute, knowing just one guy stands to benefit if the project gets monetized in such a manner?
andriy_koval 11 hours ago||||
what exactly are your concern of using agpl db server? It doesn't force you to make rest of your system agpl.
ForHackernews 12 hours ago|||
>we would be happy to dual-license under a commercial license
hobofan 11 hours ago||||
> If AGPL is an issue for anyone, we would be happy to dual-license under a commercial license.

You should be setting up a CLA process then ASAP if you are looking to take outside contributions.

abdullahk0634 2 hours ago||||
[dead]
cognitiveinline 13 hours ago|||
Sure, that's your prerogative, and kudos for not talking up open source. I'm not amazon size so can't use it, and AGPL is a no go for DB, don't want to be forced to open source my app because I use this!

Will await a MIT based fork myself.

jnwatson 13 hours ago||
Why would AGPL force you to open source your app? Unless you literally compile your app with pgrust by modifying the pgrust source code, you're safe. Clients aren't bound by the AGPL because they aren't derived works.
xyzzy_plugh 13 hours ago||
Here we go again.

AGPL is untested in courts. There is no definitive definition of what could be considered within the blast radius such that it would require AGPL licensing.

There's a reason AGPL is banned at Google and most sane companies. It's simply too dangerous.

You can't simply say "clients aren't bound" because it depends.

I'd rather see the BSL used here to be perfectly honest. At least it's simple.

appplication 14 hours ago|||
Oh bummer. I was really excited about pgrust but AGPL is a dealbreaker. Not for me personally, but it will never see wide adoption because it’s a banned license in most corporate environments. Lack of path to wide adoption means it’s dead in the water.

It’s weird because those who actually care about optimized pg gains are most likely large corporate customers. Why make a product targeting them and license it in such a way they’ll never use it?

This also hard blocks upstreaming any beneficial features into core Postgres.

ForHackernews 12 hours ago|||
> It’s weird because those who actually care about optimized pg gains are most likely large corporate customers. Why make a product targeting them and license it in such a way they’ll never use it?

Wait, sorry, you're asking why make something enterprise customers might pay for, and then not give it away to them for free?

guenthert 13 hours ago||||
If it is indeed 300* faster, I'm sure more rational corporations will rethink their license policy or be left in the dust.
hunterpayne 12 hours ago|||
Its 300x faster for certain tests. I could pretty easily craft tests that do this on two different systems. The author mentioned this when they talked about being able to fit an entire ResultSet into memory. That's the real trick with performance. Very few workloads are CPU bound anymore (linear algebra on the CPU for example). Almost all workloads are memory bound. So its all about moving data from memory to network, back to memory and back to network, over and over again through your microservices or DBs. If the entire working set can fit in memory, you get at least a 10x performance boost. If you have to keep even a part of the working set on disk, its a huge performance loss. And the larger fraction of the working set on disk, the worse the performance loss.

PS Learn how DBs do joins for more information. Specifically the differences between hash joins, merge joins and nested loop joins. They are basically fancy ways to page part of your working set to disk at huge performance penalties.

PPS As memory gets more expensive, these techniques get more valuable. When it gets cheap, they lose value.

jacquesm 12 hours ago||||
License policies are made by lawyers not by programmers. And their competition will be in the exact same boat (different lawyers though). AGPL is so toxic that it tends to be checked for during M&A processes so even if the current batch of lawyers is ok with it there is a chance that a later batch of lawyers is not. Given that the target audience for this project is the larger companies you are going to end up with a very nice project and zero actual users or you will end up with AWS et all stealing your work. Databases are very hard to do successfully commercially, at a minimum you should dual license them (AGPL for 'home' use and commercial licensing for parties that will want to buy the upside but they'll demand support and other stuff besides).
hunterpayne 12 hours ago|||
This is the case when you have your own datacenter. This isn't as big a problem in the cloud. There are ways to write licenses that prevent cloud providers from stealing while allowing customers to use the software and being required to pay for it. The problem with the AGPL has to with its viral nature, not its provisions to prevent cloud vendor theft.
nz 7 hours ago|||
A project can probably use the EUPL instead of the AGPL (EUPL is to AGPL, as MPLv2[0] is to GPL). Basically, EUPL is file-based, not project based, and so it is not aggressively viral. You can use the EUPL code any way you want, as long as you make the original code available, plus any modification to the original files.

[0]: With Exhibit B, which prevents relicensing to GPL. It is also analogous to CDDL.

xyzzy_plugh 13 hours ago||||
They could simply spend a few months and a few million tokens and get their own port, no?

I doubt even 30000x faster would prompt a policy change.

superb_dev 12 hours ago|||
Are you suggesting the AI just rewrites the whole thing under a different license? There’s no way that’s not more dicey than the AGPL license.
jacquesm 12 hours ago||
That's exactly what they did here, I don't see the difference.
hunterpayne 12 hours ago|||
This is the difference. This guy took an existing source base, had Claude find specific bugs, then had Claude fix a specific bug which was then reviewed by a person. We also don't know if these changes introduce new problems yet. You are suggesting letting Claude write an entirely new source base. That's light-years away from what happened here.
qlte 4 hours ago||

  > light-years away from what happened here
What? This project is literally an LLM port of Postgres from C to Rust. The name "pgrust" is a hint, if somewhat subtle.

There is also zero evidence a human has reviewed a meaningful amount of the code generated for the port, let alone the original C code from Postgres, given the size/scale of the project and the how quickly it was produced.

yifanl 12 hours ago|||
They did it on permissively licensed code would be the difference.
hunterpayne 12 hours ago|||
No. This is system code. You let an LLM loose on it, it probably fixed 20 bugs and introduces 200 more plus 5 different performance regressions. In all fairness, your average app programmer would have the same problems. That's why it takes so long to learn to be a system programmer and why it takes so long to do anything on a systems source base. For reference, systems are OSes, DBs and compilers (although compilers are very different in many ways).

Also, you can successfully sell a systems project that is only 10% faster. 30000x faster and they are throwing illegal and debauched things through your window to get access to your improvements.

skinfaxi 11 hours ago|||
> No. This is system code. You let an LLM loose on it, it probably fixed 20 bugs and introduces 200 more plus 5 different performance regressions.

What does it being system code have to do with anything?

hunterpayne 9 hours ago||
The brittleness and difficult of writing the code. The extra complexity. The cost of bugs. And ease of creating performance degradations. If you are even asking that question, you probably don't have the experience to even have an opinion on this.

Most systems code has to be "canonical". That means it can't be improved. Imagine the most difficult PR review you have ever had. Now give the reviewers OCD and an extra 30 IQ points. And every single code change has to live up to this standard. That's what it has to do with this.

skinfaxi 4 hours ago||
> The brittleness and difficult of writing the code. The extra complexity. The cost of bugs. And ease of creating performance degradations. If you are even asking that question, you probably don't have the experience to even have an opinion on this.

You didn't express anything unique to systems engineering.

derdi 8 hours ago|||
What do you mean, "No"? Letting an LLM loose on Postgres is exactly what happened here: https://malisper.me/pgrust-rebuilding-postgres-in-rust-with-...
kornelijus 11 hours ago|||
So, keyword 'rational', I'm not sure any sufficiently large company is a rational actor.

Yes, at [tech corp dayjob], any dependency is likely to be banned for arbitrary reasons if you bring it to the attention of the wrong people. It doesn't have to go against any of our policies e.g. don't mention anything with GPL in the name around the "risk" people. In fact, do not ever talk to the "risk" people and hope they don't talk to you.

Latest news: Apparently, devtools are a legal risk. Basic reverse-engineering of client-side JS is now banned.

The delusions really seem to scale with headcount.

karlmush 12 hours ago|||
AGPL seems like the right choice to me. I’m tired of companies like PlanetScale taking PostgreSQL, building a business on top of it, and then acting like PostgreSQL is theirs to control.
samlambert 11 hours ago||
we have not once claimed postgres is under our control. i don't think you understand how open source works but thats ok.
wrs 11 hours ago|||
Good news, we now know up front what an independent port would cost, and it’s not much. So no reason AWS, Google, and friends couldn’t bang out their own port if they want, binary-compatible with this one.

What we don’t know for sure is whether there is any copyright for LLM-generated code. The license might be irrelevant!

ognarb 15 hours ago|||
2 commits in the repo both generated by claude. This is AI slop, I wonder where you see good momentum?
Whitespace 15 hours ago|||
main indeed has two commits, but it clearly states the location of the rest of the commits, so I wouldn't be critical of main itself.

  hey claude, do a breakthrough
  You can find the actual git history at the v0.2 github tag.
  
  Co-Authored-By: Fable <noreply@anthropic.com>
Now we see https://github.com/malisper/pgrust/tree/v0.2 has almost 6000 commits in it, with the very first one on 2026-07-02. That's a lot of token momentum!

It's easy to claim AI slop nowadays, but you should still mistrust-but-verify.

f311a 12 hours ago|||
What's the reason for it? Does not make a lot of sense to keep all the commits elsewhere
malisper 11 hours ago||
It's a reference to the prompt that found a counterexample to the Dinitz-Garg-Goemans conjecture

> "do a breakthrough and find a structured counterexample"

refulgentis 10 hours ago|||
What do we mean by "easy to claim"? It is written by AI. The 6000 commits are by Claude.
esafak 15 hours ago|||
4K stargazers in a week.
busterarm 14 hours ago||
Everything around Rust is political, so the license choices are also about political statement.
johnsonjo 14 hours ago||
Most official Rust projects are dual MIT/Apache licensed by convention [1] (and most Rust libraries from third parties I've seen that are open source MIT follow the MIT/Apache dual license), so seems like this library shouldn't just be AGPL for a typical political choice of a Rustacean?

[1]: https://rust-lang.org/policies/licenses/

busterarm 14 hours ago||
official projects are usually run by sensible people who want to do things and aren't leading with ideology.

We're literally talking about an "X but in Rust" project already...

borplk 11 hours ago|
I'll take the 300x slower non-vibe-coded pg, thanks!
refulgentis 10 hours ago|
They disabled Postgres parallelism to benchmark too. Sigh.
malisper 10 hours ago||
We disabled parallelism in the blog post for demonstration purposes. The 300x slower refers to the clickbench numbers[0] where parallelism is enabled

[0] https://benchmark.clickhouse.com/#system=+liH|pgrs|gQ&type=-...

refulgentis 10 hours ago||
Who is "we"?
postgresperf 8 hours ago|||
The pgrust team asked me to look at their results on a review system, and I confirmed the ClickBench speedup there. Regular PostgreSQL is really terrible at some of these queries. Unfortunately fixing that is hard to do in core itself because columnar storage lives outside of the main tree, and some optimization problems only show up when layered on columnar.
malisper 6 hours ago||
^For context, this is Greg Smith, the author of Postgres 9.0 High Performance[0]. That book was my first introduction to Postgres

[0] https://www.amazon.com/dp/184951030X

malisper 10 hours ago||||
Me and Jason, the two people working on the project
booksock 9 hours ago|||
hi
More comments...