Posted by JohnBerryman 3 hours ago
It has the advantage of speed and the confidence not being hallucinated.
But LLMs start to generalise on the pattern, rather than the classification that you want the more examples you have to train on.
LLMs start to break down as well the more classifications you have. Laya (Open source paper Jev is based on) even mentions that over 20 classifications and it starts to fail rapidly.
20 is around the level of sentiment analysis or minor intent routing. There are cheaper, smaller and easier ML models for that level of classification.
Or they can even offer it as a standalone API if deemed worth it.
1) It's very cheap and fast - you provide one input and many potential classifications, and the compute to ingest the input is shared.
2) It generates structured output natively - guaranteed to be correct
3) It's output probabilities are calibrated to actually mean something
OpenAI, or anyone else, could certainly replicate it - there are already articles guessing how Jev achieves its "parallel" classifications, but it seems the AI companies need to decide are they in the business of providing intelligence/tokens, or are they in the application business trying to compete with all their customers (not that Jev uses OpenAI).
Don't fall for marketing BS so easily.
Jev can output drastically different probabilities if you simply reorder the list of choices. And Jev's "confidence" output is fake/redundant - it's just a formula applied to probabilities, it conveys no additional information.
I bet they will eventually "fix" (read hide under the rug) the ordering problem by ordering the list on the backend before feeding to the model.
If it really matters to you whether whether some business-specific classification confidence is above/below some specific threshold (vs just relative order), then you'd be better off training or fine tuning a custom model for that. Maybe that is something that TypeSafe are planning to also provide?
> 2) It generates structured output natively - guaranteed to be correct
It's not guaranteed to be correct: it's guaranteed to be _formatted in a particular way_. You can get the same thing with grammars on any LLM.
Jev and Jev-like models have other advantages, but I feel like people forget grammars exist for LLMs.
Is this actually true ?
the underlying data set needs to be representative
Reading logits is the cornerstone of ML. It’s almost like many of the people reporting on and fawning over AI have no technical background and never knew about ML classifiers or calibration.
Would it be intesting/useful to use jev to generate a block of text like LLMs do ?
Like asking it to pick the n + 1 word given the starting text (using it's choice primitive), but also asking n + 2,n+3 and so on at the same time.
Would it give coherent or useful results ? Or does the fact that it computes it "all at once" means it cannot make one of it's answer influence the other ones ?
They certainly have the token budget for it.
Wouldn’t be surprised if every single AI house spins up a copy
But like they usually also have an embeddings endpoint
Open weight classifiers and open weight LLMs will be burned onto silicon cards in a few years after the models begin to stabilize. They'll be in PCs and laptops. That's going to capture a HUGE chunk of the market.
If you need more horsepower, you'll rent the same silicon safely from AI services cloud providers without handing your data over to Anthropic and OpenAI.
I think demand for tools that are more tailored for this type of integration is high. I don't really understand why Jev is supposed to get my company's decisions right more than an LLM, but regardless of the tech I think people are just excited about the possibility of iterating faster, more explainability, higher-level tools that are specifically created to help hone classifiers etc.