Posted by dev-experiments 1 day ago
You can train it in under a minute, and it will work perfectly well on embedded devices.
Small LLMs are good choices for text classification in two cases:
- If you next to provide in-context examples and classifier based on them.
- Your classification goes beyond simple subject-type classifiers. For example, multiple choice question answering is classification where small LLM will work but traditional ML methods won't/
https://github.com/thelgevold/fine-tuned-classifier/blob/mai...
In summary: Using logistic regression actually improves accuracy, but also performance during both runtime and during training.
Do 5-fold cross validation, maybe stratified.
You can even get fancy and do things like active learning with the llm taking the role of the human annotator and sending in trial statements (and you can use a cheap one for larger gen and a more expensive one for the classification).
I’d be interested in seeing how well LLMs work with writing things like code for what snorkel AI used to have (there was open source code a while back that I assume is still around somewhere, you wrote code that was a low quality set of classifiers and it trained a model around those)
Trains quickly and classifies speedily on modern hardware.
Had a lot of fun doing stuff like this years ago, before LLMs were a thing.
More details here: https://www.teachmecoolstuff.com/viewarticle/using-logistic-...
- Zero-shot encoders like tasksource or GliNER
- Natural language inference: https://huggingface.co/blog/dleemiller/nli-xenc-ways-to-use
- GRPO training
- GEPA prompt tuning Qwen 0.6B (or GEPA, then GRPO)
- Use an embedding model and train a classifier (MLP, logistic, svm)
- Use a larger LLM to generate a synthetic dataset (beware of lack of diversity, mine "seed text" from real sources first)
- Synthetically generate "hard examples" where more than one category may be valid and DPO tune your preferred responses
There's even more options still, especially if you go further back toward more traditional methods. Static word vectors like GloVe or fasttext (optionally more modern equivalents like WordLlama or Model2Vec). Then there's sklearn-style stuff too. Those can be really small/fast but have more accuracy-level tradeoffs.
Can this specific failure mode be solved by providing a grammar that the output must adhere to? (Not sure if Qwen has this feature, it's used for eg. to ensure the output is parseable json)
It's something that is implemented by the thing that runs the model - eg Llama.cpp - rather than the model itself.
Note that it is hard to make work if you turn thinking on because the grammar gets complicated quickly (I don't recall if Qwen 0.6B can do thinking).
I guess the only hard constraint is to not have backtracking, right? To not waste previously emitted tokens
But actually getting that grammar right as well as actually making it work with the correct Jinja template to correctly enable thinking mode and parse it out was a lot more work than I expected.
The whole reason why embeddings work so well is because they encode the underlying meaning of the texts
Cool write up! Really appreciate it but incidentally how does this categorization help you get better retrieval results?
I wonder if one could build a 'mixture of experts' at the model level that leveraged a variety of small models "within" a larger model...
What if the question crosses categories?
also, you could stick a classifier head on a BERT model as another option.