Posted by softwaredoug 3 days ago
He clearly didn't know enough about vector embeddings.
It's the same for so many things:
- reading documentation (what do I expect this function to be called?)
- finding clothes in a shop (something long-sleeved and light)
- picking the fridge for dinner
- finding a book in the library...
so many analogues where I'm not coming cold to a choice.
Worse is when you don't know whether the answers you have are totally wrong.
The entire problem of search is that the user has the wrong data and wants to use it to receive the correct data. That was the start, not the state we’ve ended up at - it is unironically how we got to LLMs.
This method is sensitive to the thresholds (what is the maximum distance between embeddings for them to be still considered part of the same semantic group), so I run it all in an agentic loop where an agent tries different thresholds and clustering algorithms until it's satisfied with the result, plus it may deduplicate some groups.
I run it all on self-hosted hardware, so it costs nothing to leave it running for, like, a night, and as a bonus, none of the corporate data leaves the office. I think a rigid set of manually created classifications may not capture all the possible classifications that can exist. Needs a review by a human, though.
We had a similar problem where you can literally millions of email that we were pretty sure came from only a limited set of bad actors.
We first started classifying emails into buckets by From, mailserver relay chains etc as that's all we had to to go on.
Over time, those buckets got linked to spammer signatures and then we narrowed down from there.
Fascinating to see this happening nowadays with LLMs.
But honestly, it only works for common knowledge that's already in the LLM. If the target document contains very niche or private information, then the hallucinated answer's embedding can be even farther away than the query's.
Even better is to search the corpus first with like naive BM25 / embedding search, aggregate over top N to get most representative categories, then have the LLM categorize in that set.
Isn't this begging the question that the hallucinated classification will be more selective with respect to the real schema than the query itself? What would the dot product of <E(search query), E(schema)> have given?
Even if that is too vague, smaller LLMs are capable rerankers; return the top N matching true categories and ask for a contextual ordering.
I've found, though, getting it in the language of the vocabulary has generally improved performance.
Further, when searching for "blue shoes" you want to separate the color from the item type. So its useful to have a dumb LLM do this for you. And with the LLM in the loop, its further useful to get it into the language of the taxonomy to improve embedding retrieval accuracy.
There are of course many ways to skin the cat here :)
But no classification is perfect. In search in particular, you will also want to have places for manual intervention for high priority queries.
Additionally you could experiment with a reranker instead of an LLM or after reranking take top-3 results and then feed to LLM as input in order to reduce input token costs.
Putting all the labels into the LLM is super expensive per call when you have millions of items to classify.
You can't reduce the number of labels becasue they are correctly organizes/structured. This class of problem exists in many different domains.
This problem has also been solved for 3 decades now.