Marketing seems to have dropped the ball on that. I doubt anyone's picking it for the +2% higher points, but being able to answer based on a PDF or call is groundbreaking.
Jev is still faster and cheaper for many of the common text uses.
Google's marketing consistently drops the ball on every opportunity so that's not surprising at all
But maybe this is one of those things that's only obvious in retrospect? I'm sure people were trying something similar years ago with grammar-constrained outputs (for example to only output a simple json structure), but you can't get probabilities out of the model that way. Which makes me wonder: how do these models calculate probabilities?
> how do these models calculate probabilities?
The short answer is: more and better data.
The long answer is: probably the same way any other classifier does. Take your ground truth probabilities and your predicted probabilities. The gap between those is called cross entropy, which is used to push the model towards the correct probabilities.
The definition of "correct probability" here is usually "p=1 for the ground truth label, p=0 for all others." Which seems like an intuitive approach but if you actually think about it doesn't trivially imply the model would learn sensible distributions under uncertainty.
The various decision APIs don't give you that, and would themselves need to be recalibrated with constraints in feature-representation space over the applicable reference class, as with https://pypi.org/project/reexpress-sdm/.
So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.
I’m not sure that’s true in real cases.
I would expect that in distribution tasks have quite consistent outcomes with variations on phrasing. I would expect that to be much less true when tasks are out of distribution. So could we randomly adjust phrasing and see how much the results change - looking less for truth but consistency?
nit: it's trained on finitely many samples.
> So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.
Huh? That's not counter to anything I said. All I said was it's not trivially implied.
Chain of thought was that way. DeepSeek came out, and not long thereafter everyone had adopted chain of thought.
https://darioamodei.com/post/on-deepseek-and-export-controls
> In 2024, the idea of using reinforcement learning (RL) to train models to generate chains of thought has become a new focus of scaling. Anthropic, DeepSeek, and many other companies (perhaps most notably OpenAI who released their o1-preview model in September) have found that this training greatly increases performance on certain select, objectively measurable tasks like math, coding competitions, and on reasoning that resembles these tasks.
I remember adding it to GPT-3.0 and getting a big boost in accuracy. It was like magic.
And then we would handcraft thoughts and inject that into the prompt so the model thinks its owns thoughts and would follow instructions much more precisely.
This was well before reasoning models, even before chat models.
It was a short-lived time though. I think only a few months before Deepseek really RLed the reasoning which produced much better reasoning than just prompting a CoT out of the model.
The natural path of things, I suppose, is those that have a soapbox/distribution will just get larger...
You're leaving out the UX (DX, developer experience) in this.
And the better zero-shot performance when you don't have a dataset yet (could argue this falls under DX).
Not to mention the faster time-to-decision and order-of-magnitude cheaper costs.
The whole system is foreign to us, and makes us rethink what "intelligence" actually is.
Today developers without any AI/ML skillset can prototype a fancy new feature with LLMs, what used to require collecting and curating data to train a pipeline of ML models then serving them in production.
The problem comes when it's time to take the new feature into production. That $20 cost with prototype/beta workload is now $30k per month in LLM costs, and those hallucinations that never happened on their 200 test samples is happening 600 times per day.
They are then stuck, because they lack the knowledge and budget to build and run the production ML pipeline properly, the infra stack necessary to handle production workload. Management also doesn't understand why they have to invest in rebuilding something that "seems to work" with "outdated methods".
These system one or decision models are a step up from LLMs when it comes to this usage. Problem is that they are still difficult to work with (limited in expression power) for many application logic. Despite the initial popularity, I hear about a lot of frictions refactoring LLM based backend with them, and multiple cases of production rollbacks to LLMs already.
I think system one models is a sensible paradigm, but there is a lot more work needed above the model layer to make this work.
Roughly speaking, has AGI actually been achieved two years ago and we go around don't noticing that?
By unprincipled methods. To calibrate confidence in an answer you need a task specific calibration set. You can't pre-calibrate for tasks unknown ahead of time, which is the case for most uses of decision models.
Hell, human thinking is unprincipled in basically the exact same way.
I guess given the huge advances in LLM since then, these models probably achieve much better accuracy on a wide range of domains with lower effort compared to what we were doing, which makes it compelling as a product. And yeah, the models probably were already good enough a few years ago too. Other factors include how easy it is to release products at high speed now, and the desire in the industry to reduce token spend.
Decision models are really not much different than that, but (usually) scaled up to bigger causal models.
If you are looking to do Jev-like things with smaller models, check into NLI.
You can do it manually or use ready made libs. I know there's one in python on top of vllm that can do something like this.
Every time I see someone saying something like "isn't this just structured output?", it's because they don't understand that probability calibration is one of the most important parts of the product, especially to understand uncertainty.
I think Jev did a pretty good job on calibration. I wonder how the fast-follow competition from OpenAI and Gemini do.
Once a model goes through instruction fine tuning and RLVR/RLHF the log probs decorrelate from what's ground truth (given the propensity of the data seen during training)
Because it's especially parsimonious with token use.
Which classification tasks aren't?
I like that it’s fairly deterministic, with a bit of variation. Think very low temperature rather than perfectly repeatable. I just didn’t want the randomness that comes from token by token execution even at low temps.
I’ve also used it for “does this contain an entity”, and then I run an LLM to extract the entities. Saves on the LLM cost.
https://chromestatus.com/feature/5155080092385280 https://evergreen-labs.org/products/credibility-analyzer
Guardrails - check the text against a set of guardrail checks (questions)
Search - Given a search criteria, I first have an LLM generate a query that would go to the decision model. This query is a rubric that scores each document. The highest scoring documents become the search hits. This is fast and cheap, so you can use it as a cross-encoder for your entire set, instead of just as a final re-ranker.
We use it for our critique looping within our own harness.
But decision models can do things like write SQL queries, if you're willing to put a little bit of trickery upfront.
My team's writing a paper about it for ICDE 2027 as we speak (as a follow-up to the DICE demo/paper from VLDB 2014).
But my Linkedin post[1] has a quick summary of the idea, without explaining everything we do there.
Evals. is this hitting the criteria or not? (Test case passes or fails)
As an example: I could have something I'm building fail. If I take a picture of it, can I see that it's
1) a false failure
2) salvageable
2a) decision-tree for *who* to send it to for reworking things
As I think of the above decision tree, I see some problems where I'm just asking myself, "why wouldn't I just use <insert a lot of options you can probably guess here>?".
I'll be honest, I think right now decision models are in a "solution looking for a problem stage", but I think there's opportunity still.
However, it can accept Audio and Video as input as well and for those specific cases it can be extremely useful.
I'm not even sure what all is going on inside it anymore, because I just add new features when they popup, and tell Claude to do it, and it let's me know when it is done, and every new "input feature" can run against the entire catalogue of files.
Eventually I'll add facial recognition to the photos with one or more people, but I've not felt like delving into that can of worms. I tried it years ago. I've contemplated just training my own model off of the few 10s of thousands of images I have to automatically tag them in a single pass since I have everything labeled already.
Out of the hundreds of questions, most of them could even be different variations of the same question to capture more nuance than what is normally possible in a single multiple-choice.
Being able to get a fast, high quality answer to the question "does this warrant interrupting a human?" or "does this look suspicious?" was a game changer for us.
I was also shocked to see that after a week of heavily experimenting I did not even burn through 5 USD!
Wish they provided more comprehensive inputs to the model's capabilities and comparisons.
Why does this need to be a separate thing?
It comes with a bunch of pre trained classifiers, and you can use the tooling to train your own
There are also integrations with workflow tools like n8n