Idk, your limitations section sure makes it seem less drop in and less general than Jev. Like the point here isn't your ML aptitude it's how easy is it for developers to drop this into a product and use it.

I'm more than capable of training a bert classifier in fact in 2019 I had trained many custom berts and was running them on hundreds of millions of documents a day.

I don't want to manage GPUs / CPUs now. I don't want to maintain my corpus and retrain as my product's data distribution shifts. The list of things I don't want to do goes on and on and on. And I'm happy for them to be someone else's problem.

I do just want a reasonably good general classifier served to me with a great devex and calibrated confidence scores to help me figure out when to fallback to another model.

I’ve tried it versus Jev and I got significantly worse decisions. I hosted it on runpod nvidia t4.

I wanted to classify business b2b vs b2c and business model. Am I holding it wrong?

>Zero-shot vs. Fine-tuning: Out-of-the-box base models score ~0.35 on the typed-decisions benchmark (near random). The 0.766 score is achieved by fine-tuning on the benchmark's train split. Treat Laya as a fast foundation model to specialize, not as an omniscient zero-shot oracle.

This should be way up in the article. Fine tuning is a pain, requiring it for good results put Laya in a whole different category vs Jev

Quickly reading the article, one notable limitation seems to be that these checkpoints are 512-1024 tokens context size models, while Jev is seemingly 32k.

That's a pretty big limitation, I would argue, unless I'm misunderstanding and it can be worked around easily somehow? I'm surprised it isn't surfaced more prominently in the comparison.

LLMs being described as system 2 thinking here is a semantic shift I have not encountered before.

LLMs are also a deep learning approach. Output, as slow as it is, still comes from weird latent spaces. In AI I always took System 2 to map more to symbolic approaches, or at least when explaining symbolic AI to someone who has heard of deep learning thinking fast and slow was a good comparison to draw on.

Distribution > Creation

Huge omission. This requires fine tuning.

> Zero-shot vs. Fine-tuning: Out-of-the-box base models score ~0.35 on the typed-decisions benchmark (near random). The 0.766 score is achieved by fine-tuning on the benchmark's train split. Treat Laya as a fast foundation model to specialize, not as an omniscient zero-shot oracle.

Love it. I was really surprised to see the traction typesafe got in the first place. I had built something similar a year ago for a client and thought it was nothing groundbreaking. The client bought it, still uses it and that was it. I had also spent considerable time training and fine tuning zero shot NLI classifiers. Anyway, after typesafe was launched I decided to start building this open source library - https://github.com/deepanwadhwa/OpenDecision . The context length for the underlying model is 8k.

The unfortunate true is that getting even the best work in front of an audience is often much harder than solving the problem. Is uploading a paper to arXiv enough to expect the work to be recognized and cited? Unfortunately, it rather is not. arXiv is an open repository which includes plenty of not reviewed and not officially published papers. In a popular field such as machine learning, the number of arXiv papers is overwhelming. Expecting that some machine learning expert will stumble upon an arXiv paper and recognize its value is wishful thinking.

I'm not a researcher, but long time ago I had an idea of a new, seemingly interesting attack on TCP. Having some free time between jobs, I wrote a paper about this, created a proof of concept and decided to send the paper to USENIX Security. I got back two reviews, both in rather positive tone, but rejecting the paper on the grounds that it shows only individual steps of the attack, but it would be much stronger if it showed also the attack working end-to-end. At that point I just uploaded the paper to arXiv and called it a day. I've put a lot of work into that paper, but not enough, I don't consider it properly published and I don't expect anyone to cite it. The paper failed the peer review process and I didn't put the work to improve it further.

Jev will continue to do well because people don't actually want to host their own models. The average customer just want an always-on pay-per-use API that has social proof.