First release people are getting more balanced at least. Less 'it's gonna change everything' to 'might be viable use cases'. If I have to look at one more video high lighting google flights, I might loose it though.

Looking at the logprobs on tokens works for the local models, but not on the frontier ones. It's been more or less broken since GPT-4o for example. I wrote about it two years ago: https://medium.com/data-science/9-11-or-9-9-which-one-is-hig.... Also, I've done some work in estimating confidence and on rubric evals using the same method, and you actually get better correlation to "real confidence" by just getting the LLM to say it.

I wonder if this could be a good stepping stone to write a local prompt router to optimise what model get what prompt. I.e. if the prompt is just a lookup, send it to haiku, if it's reasoning, send it to opus and if it's implementation send it to sonnet.

Could someone explain how Jev is different from using any old model and constraining the output to "My choice is a/b/c..."?

This is a system two model, and not a system one. To get the performance of Jev, and you want to run locally, use Laya. It is up on Hugging Face.

Maybe someone can explain why RL is even needed for post training with Jev? We have supervised labels.

I guess it's due to the calibrated decision part (and that's what LLMs tell me).

But I figure some supervised classification post training would still improve the model.

Am I missing something here:

p(y = next thinking+decision token | x = question) != p(y = next decision token | x = question)

The former is what LLMs are trained for, the latter is what Jev was likely trained on (likely used thinking alignment as an auxiliary loss, but not explicitly included in the probability calibration).

We have Jev at home

i love that people are trying to make OS jevs but what is the point of doing all this work and not ask your coding agent to do a little benchmarking. selfishly want an open weight model to beat jev here

just found this one https://huggingface.co/spaces/multimodalart/jev-decision-ind...

I missed the hypewave so can't say a lot about Jev, but the double standards are entertaining:

About Jev:

> We didn't train a model with Reinforcement Learning for Calibrated Decisions (RLCD) to calibrate the decisions and probabilities (even though they are not always correct).

Only 99% correctness! Borderline unusable!

About their model:

> It classifies: it gets a prompt with choices and outputs probabilities.

You want numbers, it gives you numbers! What more could you want?