Every major AI shop has a ton of in-house classifiers already, big, small, generalist, specialized. Some are used in inference pipelines (e.g. safeguards), some are used in data preparation, training, various one-off and intermediate tasks etc. Offering them on a public API doesn't always make business sense. I don't see much substance to this buzz, looks like people that are new to all this are discovering that classifiers exist, they are more efficient at classification, and many tasks commonly done with generative models are classification in disguise. Which is not bad at all, a fresh look at their use is great to have.

This article is extraordinarily hard to read. It’s tummelvisioned on OpenAI and things like tool calling which are only relevant to the extent that llms have been tuned to make relative choices, but this applies to all LLMs. Also, some really outdated references. LLM written, perhaps?

Also, moat discussion is the lowest form of discussion. I don’t care if jev has a moat. Did it get the interface right? What other past ideas have we overlooked that if given some love, could kick the door down like jev did?

Really silly stuff.. people wanting to talk about moats when there’s no castle. Moat talk merely projects the illusion of being engaged but, much more often than not, it’s hollow engagement.

I find it unlikely. OpenAI is all in training models with reasoning with RL, and Jev-like models are the total opposite. They are made to not reason at all to be fast. If you want to add reasoning on top, you might as well use a conventional LLM because you lose the price and speed benefits when you output auto-regressive tokens. I don't think OpenAI will even bother with this.

> My main assumption is that Jev is using something quite close to a conventional large language model. As evidence of this, Latent Space reports that many of the early clones are indeed LLM-based.

Not proof that this is the case with Jev though. It might use non causal text encoder for the state, which could make sense given that it's very good for its price.

There is one benefit that Jev has, that it is not OpenAI and thus it's probably less likely to steal your own work.

Isn't this more and more likely on all shapes of model evolution? The providers will all copy each other.

And in this case I hope its true. I've been classifying a lot of email and while OpenAI `text-embedding-3-small` has been very helpful for fast and cheap embeddings, initial tests with Jev are very promising and much more ergonomic.

I put more thoughts here: https://housecat.com/blog/classifying-email

For context on what "Jev" is: https://news.ycombinator.com/item?id=49717558

Rather than focusing on OpenAI in particular, let's just say that there are many smart people at other AI labs and if it seems like it will be popular, this technique will probably be copied. What would prevent them from adding another API?

Hopefully there will be some decent benchmarks and gateways for switching between providers easily.

If OpenAI releases something similar to what Jev does, then that would be like admitting defeat. Their whole spin is AGI and world ending danger. Why would somebody with an AGI at home make something like Jev which is intended to be a part of some SW the AGI is going to replace anyway.

    > so why invest into a more complex solutions

comments are pretty weird here, there's no real moat to what jev is doing, it is certain that frontier labs are going to release their own jev and there are even open source alternatives (although nowhere near as accurate as jev).

so maybe typesafe's real plan is to front run and releasing their own new models for some time until they can get acquired which seems to be the only rational objective

I think the article is part right and part wrong.

The right part: autoregressive LLMs are indeed generating “probabilities” (scare quotes very much intentional). During pre-training and any SFT steps, those probabilities are nudged toward the probabilities, over the training distribution, of the next token conditioned on the previous tokens. (This is an explicit property of most training recipes: KL divergence is a “proper scoring function”.)

So if you prompt with “Paris is a city in ”, the next token probabilities estimate the probabilities over the input distribution that the next token in the sentence is the first token of France or of something else.

But there are huge caveats:

1. That is not at all the same thing as the probability that Paris is France under any distribution that you care about (the population of the various Parises, for example).

2. None of this necessarily usefully applies to RL or, as the article discusses, tool calling. The output probability of a tool call is not some Platonic idea of a probability that the input is worthy of a tool call. It’s a the result of a training process that tried to teach the model to be useful and to achieve its goals.

3. I suspect that reasoning makes this all much worse. Suppose that you prompt with “a help desk user with IP=a.b.c.d says they’re ‘in Paris’. What country are they in?” The model has been trained to generate a reasoning trace, which may well start with “let me think of where Paris could be. It could be in France or in Texas etc. The user was speaking English…” See the problem? The model is reasoning well, but it reasoned “France” before “Texas”, so the logprob for France was probably higher than “Texas”. At the end of the reasoning trade there will be an answer, but the logprobs for that answer are, at best, some representation of the probabilities of the answer conditioned on the sampled reasoning trace. And that is not the probability distribution that a Jev user wants.