Posted by ksec 7 hours ago
It started with paradeDB and pg_search https://www.paradedb.com/blog/introducing-search
Timescale has pg_textsearch https://github.com/timescale/pg_textsearch
Neon and Databricks have Lakebase Search https://docs.databricks.com/aws/en/oltp/projects/lakebase-se...
Now PlanetScale.
AFAIK all of these are implementations of the BM25 algorithm. You can just tell an agent to read about BM25 and implement it in your system of choice. Cool to see. Seems like there's still a lot of juice to be squeezed out of how it's architected and integrated into each system, but you can't help but wonder if this will lead to aggressive commodification
Planetscale (assuming they used a agentic development practice) will have pulled this off, to the level of performance that they have, because they have a team of very highly experienced Postgres developers. Their knowlage of Postgres internals will have given them the insights needed to steer the models to a plan that used the architecture as described in the post. That's not something a model can do on its own*
World experts + LLMs = moving mountains.
(* we're obviously seeing something a little different from inside the research teams in the labs. They are showing that the models, when you burn the level of tokens only they can, are able to do novel things from the models own insights.)
I am exploring this exact area of search and analytics for vanilla postgres as replicas. Guess what, the LLM came up with this exact conclusion of using ctids as docids, all by itself. It was surreal for me to read the blog above , when I hit that paragraph about ctids.
I am no postgres internals expert.
Yeah, I don't think I could tell Qwen3.8 (my LLM of choice) to study up on bm25 and then implement full text search in the couchdb instances I maintain without studying both bm25 and couchdb internals myself.
More damning for the theory might be that I think paradedb's pg_search predates the agentic coding by a few years?
They're not providing a local extension with the same performance at the time - it's only offered on their cloud services.
The local version https://github.com/planetscale/lead is mainly just for testing the syntax, it doesn't have the same perf characteristics.
Immediately rules out ever using them (though I don't currently have any problems that would benefit from that level of scale currently, have in the past though).
Postgres's license allows this but for me (personally) it leaves a bad taste.
Also it's not really "full-text search for Postgres" it's "full-text search for our hosted version of Postgres" so the title is a little misleading.
Uh, isn’t this becoming the norm everywhere ever since LLMs have been trained on OSS without credit or attribution? Why wouldn’t you want to hide your stuff going forward?
In my view, OSS is only going to move more and more towards one of two models: open core + proprietary functionality (e.g. MongoDB) OR open source + private tests (e.g. SQLite).
The license allows it but on the flip side it's vendor lock-in predicated on using something open as the base.
Fully proprietary no issue with that, full open, no issue with that, building proprietary on top of open is where the bad taste comes in.
For completeness, it's not them specifically either, the other cloud companies do similar things and I suspect in part the reason they don't open these extensions up is because the others will but then they are doing the same thing themselves.
(Disclosure: I work for a company with this business model, in part because I like working on open software)
Those that develop on top of open-source have no obligation to give you their work for free.
You’d be surprised as to the amount of open-source contributions TIN drove towards Postgres, LLVM, and pgrx. And you’d be speechless at the amount of upstream work across all sorts of open-source PlanetScale does. Postgres 18.6, for example, is better for you today, in part, because of TIN. You’re welcome.
I didn't say thank you and don't presume I would, Planetscale acting in their own self interest by improving postgres upstream isn't deserving of thanks, any more than Intel upstreaming a bunch of Linux kernel work is or myriad other examples.
Corporations acting in their self interest isn't worth giving thanks for, neither is the work of the people paid to do work on their behalf.
I don't expect my employer to thank me, I expect them to pay me, I don't expect users of software I was paid to write to thank me because I did it for the money not out of altruism towards those users.
Good luck out there! 2026 is wild times!!
> Those that develop on top of open-source have no obligation to give you their work for free.
This is a pretty narrow view of open source.
Just as a simple example, GPLv3 and AGPLv3 are both considered Open Source and, depending on how you hold them, may obligate releasing work to customers essentially for free.
I remember so well when "Open Source" branding started with Bruce Perens - all the business arguments. When you need a database, someone ELSE'S business decisions (how THEY are going to make money) never end up helping YOU. If you use proprietary extensions and become dependent on them, you inevitably will get BURNED when their business needs diverge from your needs. - They close shop - They refuse to interop with something you need - They demand that you obey their arcane rules - They rug-pull - They get hacked as only they can - They lie to you - They stab you in the back
> When one develops using open-source software they have an obligation to follow the licenses.
Obviously what follows from one of those licenses is what you say.
I’m glad we agree!
Agreed!
> They also have a moral obligation to be respectful of the work upon which they’re building.
Respectfully, completely disagree!
Open source is a license, not a moral framework. Your only duty is to abide by the license. If you want to enforce that everything built on top of a particular open source package also be open source, that belongs in the license.
We live in a society. I don’t see any harm in believing we ought to be respectful of the work. That’s all I’m saying. You don’t have to be, but it’s a better world if you are.
> If you want to enforce that everything built on top of a particular open source package also be open source, that belongs in the license.
Yes! I fully agree. The one that creates the thing is the one that gets to choose its license.
Having opened-sourced some bit of work myself, that’s a very difficult decision.
PostgreSQL built-in FTS provides a score for each row based just on the data for that row.
Relevance algorithms like BM25 take overall corpus statistics into account. If you search for a bunch of words and some of them are less common than others in the overall set of documents, documents that match THOSE words will score higher than matches for other words in your search.
That's what all of these additional extensions are providing.
- superior performance
- superior operational overhead
- no second copy of data in tsvector form
- BM25 scoring support with optimized top-k output
- runtime configurable scoring knobs
- expression-attached score boosting
- sophisticated span query support -- this is proximity search on steroids (https://github.com/planetscale/lead/tree/main/tinql/docs)
- lossless term positions
- index-answerable negative expressions (find all docs that don't contain a word)
- full document hit highlighting
- optimized exact `count(\*)`
- term expansion via any of fuzzy matching, wildcards, regular expressions, and dictionary ranges
- intentionally smaller user-facing SQL API surface
As an aside, something I personally think is cool, and I suppose you can do this with Postgres' built-in `@@` too, is that you can use TIN's full query language (linked above) against any text datum. This is a valid query:
SELECT pid, query
FROM pg_stat_activity
WHERE query ==> 'select OR copy'
Incredible? No.
You want to use this thing instead?
For example this one: it doesn't mention support for CJK languages (meaning tokenization for e.g. Chinese will resolve to one token per character, which will technically work and give results, but is inefficient). Also word stemming (databases -> database) is also missing as far as I can see, so the kind of queries where you'd expect related words to show up will be missing. Just doing case-folding and accent-folding is a bit of a functional but bruteforce solution.
Ideally I'd want something that supports:
- language aware tokenization, with ability to define the language per record. Including stemming, etc. And have useful predefined configuration for common languages (e.g. the Postgres built in one is missing many languages).
- CJK support, tokenizing at word boundaries.
- Optional accent- and case-folding.
Most solutions just seem to assume English content, I have not found anything that does all of this yet.
I've always preferred to use SQL as the system of record and then build/maintain an external Lucene index. Do we think these integral FTS capabilities are at the point where a hybrid architecture doesn't make sense anymore? How much customization exists in this provider?
The answer to your first question is simply: yes
As far as your second question, what customization do you need that you believe TIN or PlanetScale doesn’t provide? These are things we can do, with alacrity.
Having everything in one place eliminates entire classes of development and especially operational problems.