Posted by bashbjorn 17 hours ago
In my experience even structured LLM output performs poorly on classifier tasks. LLMs are trained to talk and think longer. If you don't give LLM enough space to reason it would become very dumb.
I'm not saying that Jev is way better, but that people way overindexed cost and speed.
You have built something like jev but not jev (for starters, the output of what you've built will be absolutely worthless, the whole reason Jev is getting so much hype is because the output is good enough)
A Google AI prompt says
> TypeSafe AI's Master Customer Agreement explicitly prohibits using the services or model outputs to develop a competing product, perform model distillation, or reverse engineer the service, which generally restricts competitive benchmarking aimed at replicating the model.
It doesn't explicitly prohibit benchmarking by name, but the previous terms (which seem aimed at preventing Jev being used to increase the value of competitive products) does seem to lean that direction.
That said, MsSQL had terms which prevented publishing benchmarks which compared it against other SQL DBs and that wasn't enough to prevent some companies from using it.
Why anyone would want to work for a company who thought so little of their own product that it couldn't stand up to customers using it for normal business processes is beyond me.