Posted by bashbjorn 14 hours ago
I've found that using structured outputs solves this problem much better. Instead of letting a model generate only "A", "B" or "C" and looking at the probs, have it directly generate "Legitimate", "Spam" or "Phishing" or any other pre-defined option from a set of multi-token sequences. Behind the scenes it boils down to something quite similar, but you're not running into the risk that the model actually wanted to say "A phishing attempt seems likely, so answer (C) is correct.", which would lead "A" to have the highest probability in the first token. You can even use a reasoning budget this way either via inherent reasoning or a free-form part preceding the remaining output structure. You can also have it assign probabilities (either in words or numbers) using more complex output structures, but I would not rely on them much more than the token logprobs (they can still be quite good though).
I got this technique to work extremely reliably last year. However there were a bunch of caveats: 1) Firstly, you must institute a check that the multiple choice tokens dominate the output distribution. They should sum to 95% or more, ideally 99%, or the LLM is not following instructions properly. This is also the problem with constrained decoding - if the LLM really doesn't want to output a valid answer, the one you extract will not be high quality. 2) You need to ask it multiple times, permuting which option corresponds to which letter, and average the results. LLMs are surprisingly biased towards picking "A", especially if they're otherwise not sure. 3) For the same reason, performance improves if you frame the prompt as if it were the middle of a quiz. "Question 1" carries baggage that "Question 12" doesn't. 4) You must be exceedingly careful with tokenization.
But when all was said and done, I got a general purpose A/B classifier that gave high resolution quantitative output for the cost of a couple dozen tokens ingested and a couple inference passes.
The whole point of my argument is that neither is good, but from a technical perspective logprobs is probably the worst unless you train a model on specific outputs. In which case you'd throw out the generality again, so when I think about it more, it's actually the worst overall. In my experiments, having the model simply assign "high" or "low" probability in a structured output generally performs best. You can try numbers, but you will never get anything close to what you could expect from traditional ML. And most certainly not from logprobs.
GP pointed at a causal explanation for this: almost every sentence in English that's a statement will start with "A" or "An", so "biased towards picking ''A''" will include most attempts at saying anything long-form for any reason.
Meanwhile, the bias could be as much as 70% in favor of A in ambiguous cases - a signal completely drowning the <1% inclination to violate the format.
Not nearly as sophisticated as myself who would mutter "When in doubt - Charlie out" before marking C.
Also, Jev/laya do it in one forward pass, for multiple questions about the same state, rather than multiple passes for one question about that state. Well, for the usual multilingual configuration, two forward passes through different small models for laya, but that's because one is the router which chooses which model should do the real work, but still.
I contribute my experience here only because I've seen a lot of chatter lately about doing exactly this sort of thing, and I thought I'd share how I made it work for me. There are a lot of ways it can silently fail and give bad numbers if you aren't careful, and I wouldn't want people to think it doesn't work just because they used a vibe coded GitHub project from the last 48 hours that doesn't take these things into account.
I ran some tests using GPT-4 to do some basic classification a couple years ago. On ambiguous options which had to be escalated to a human, the LLM would regularly output something like a 99.8% probability, compared to 99.99% for a correct answer.
https://til.simonwillison.net/llms/llama-cpp-python-grammars
Prompt part: "What is better, toast or bread?"
Incomplete answer part: "The answer to this question is "
and then have the LLM finish the answer. I did this with subtitle translation using llama.cpp (with Python) and had great success. Just past 5 already translated subtitles as the incomplete answer, and the LLM infallibly just continues to translate. No markdown, and usually no talkback if the subtitles contain nasty subjects like bioweapons or nuclear stuff. It just works.
To completely squash the issue, a few cheap LoRa iterations will do the trick just fine.
I think we can all agree that Jev is not rocket science. It's a good idea executed well, with marketing that might have been a tad too bold
And further down "TypeSafe computes confidence from how the probability is spread across the options. All of it on one option gives 1.0; the more evenly it spreads, the lower the confidence. This demo uses (3 × largest probability − 1) / 2 to approximate confidence for three options."
So while we don't know the exact formula they use, it is just a function over the probabilities
I am open to the argument that this does not work well if you just plug in a qwen model instead of a model that is trained to output more statistically useful token distributions
It means confidence is just a converted max probability and not an independent signal.
we agree then, that is the entirety of my argument. Getting a deep net especially one that is anywhere near even SLM size to be calibrated is tough, especially across domains. They claim calibration across a variety of datasets which is interesting.
Actually to me it sounds it could be benchmarked if this kind of effect exists in the first place.
There, I fixed your problem.
Another trick that works is to repeat the question two times: "I'm repeating the task and labels for clarity: ..."
Payroll sends you an email with a link to a Youtube video that plays a song.
Options after body:
Average probabilities:
Rickroll 0.5158 ( 51 wins)
Phishing 0.4561 ( 47 wins)
Spam 0.0281 ( 2 wins)
Joke 0.0000 ( 0 wins)
Legitimate 0.0000 ( 0 wins)
Average probabilities:
Rickroll 0.9293 ( 94 wins)
Joke 0.0549 ( 5 wins)
Phishing 0.0140 ( 1 wins)
Spam 0.0018 ( 0 wins)
Legitimate 0.0000 ( 0 wins)
EDIT
Gemma4-12B-it-NVFP4 seems way less sensitive to option/body ordering:
Options after body:
Average probabilities:
Rickroll 0.9867 ( 99 wins)
Phishing 0.0133 ( 1 wins)
Joke 0.0000 ( 0 wins)
Spam 0.0000 ( 0 wins)
Legitimate 0.0000 ( 0 wins)
Average probabilities:
Rickroll 0.9401 ( 93 wins)
Phishing 0.0336 ( 3 wins)
Spam 0.0250 ( 4 wins)
Joke 0.0010 ( 0 wins)
Legitimate 0.0002 ( 0 wins)
I am glad there is an actual reason.
Especially ridiculous is how the hacker news crowd seems to be taking these at face value…
There was one the other day with a compelling demo. But when you looked closely at it, it was feeding in the options with the word “best” on the option to pick and a fine tuned model designed to recognise that word…
P.S. I am evaluating that model for a production use case where I would have used Jev
So you might ask: how do I obtain the training examples? Just collect samples and use a coding agent to classify them as match or no match. From time to time you can add more examples to the dataset to have your concept adapt to changes in input distribution. It's all automated, but it only uses LLMs to train concept vectors, after that it works like a regular embedding model with a calibrated classifier on top. It's also 20-30x faster than Jev, free, and runs on CPU.
An illustration of how it defines a concept as opposed to simple cosine similarity: https://github.com/horiacristescu/semlabel/raw/main/images/c...
>calls an api
ok
But then at the end it says it’s parody. Maybe HN title should say it’s a joke.
you can swith to a better model for lower error rate.
Somehow the HN crowd has a bunch of "professionals" who don't care about error rates and think that a Qwen model running on a potato is frontier intelligence.
(For more realistic solution, surely someone must be working on optronics - these models just beg to have their weights cleverly etched into stacked sheets of plastic, so they can do inference for free on a beam of light.)
Non deterministic systems have furthered the "brain rot" in our industry.
Lots of people were happy to ignore the code in their "supply chain" before LLM's - but suddenly not reading the LLM's output is a problem. I get they are different but we're in the same realm.
The lack of real data on performance of what ever application that one is trying to pitch is getting appalling. It's a lot of "trust me bro" this works better hand waving. And it's getting gross.
And how do we even measure nondeterministic systems? Because if I told you that Anthropic was spending millions of dollars having 1000's of agents "pre solve" benchmarks to build into their next version of the system you would scream they were cheating. Every one is focused on the "hacking" in the hugging face incident and no one is looking why they were even playing with those benchmarks in the first place.
"Trust me Bro"...
But aren’t calibrated predictions one of the defining features?
