Do You Need an LLM for That? Classifying Text Locally

One of the things that keeps me up at night is a simple question: what is the best way to classify text on my own machine? And, now that LLMs are everywhere, do I even need one? More generally: what should I do when the environment is constrained, with little RAM, no GPU, or a tight latency budget?

The reflex in 2026 is to paste the label list into a prompt and ask an LLM. It works, sort of, and you never have to train anything. But text classification is an old problem. A bag of words and a logistic regression have been doing it for twenty years, and small fine-tuned encoders like BERT have been doing it very well for the last eight. They are cheaper, faster, and they fit in a few hundred megabytes. So before reaching for a 4-billion-parameter model, it’s worth checking what it actually buys you.

Four questions:

1. Does the number of classes change the answer? (CLINC150, from 10 to 150 intents) Listing 10 labels in a prompt is easy; listing 150 is not. Does prompting fall apart as labels are added, and do trained models care?
2. Or is it the kind of task? (VAST, then DynaSent) Some tasks are given away by their words, others are not. Which approach survives when they are not?
3. How many labels does a trained model need? To beat a prompt that needs none, on each of the three tasks.
4. What does each approach cost? Not only quality: labeled data, training time, latency, and RAM once deployed.

One rule: everything runs on my laptop (Apple M5, 24 GB of unified memory). All the code is on GitHub: driquet/local-classification.

A word of warning: this is a personal weekend project, so I kept it to the minimum. One seed, one prompt, no tuning: that’s not how a paper would do it, but it’s enough to get the gist. I’m also the kind of person who needs to tweak and hack things to understand how they work, and I’m convinced every insight in this post is already covered somewhere in the literature (a few pointers at the end). This is me finding it out by hand.

The contestants

#

One per approach, from the oldest trick in the book to an LLM trained to make decisions:

TF-IDF + logistic regression. Each text becomes a sparse vector of word and character n-gram weights, and a linear classifier does the rest. This is the floor: anything more expensive has to beat it to earn its keep.

Ettin-150M, fine-tuned. Ettin is a family of encoders from 17M to 1B parameters trained with the same data and recipe; 150M is the classic BERT-base size. 3 epochs, batch size 16, learning rate 5e-5, gradients clipped at 1.0, and at least 200 steps, so that tiny training sets still get enough updates to move the classification head (Mosbach et al., 2021). This is the textbook answer when you have labels. (Which size to pick will be addressed in a future post.)

Qwen3.5-4B, prompted, in BF16 with MLX, thinking off. I don’t use a quantized version, so that quantization can’t affect the F1 scores. The price is paid in RAM and disk (and a bit in latency), as the cost section shows. Instead of generating text and parsing it, I read the probability the model gives to each label, so the answer is always a valid label. Two prompts:

• Zero-shot: the prompt lists the labels and asks for one. No labeled data at all.
• Few-shot: the same prompt plus labeled examples from the training set, drawn at random: one per class when there are many classes, five when there are few. Every text gets the same examples, placed first, so the model reads them once and keeps them in its cache.

OneJev1.6-4B, an open decision model released in October 2026, built on the same Qwen3.5-4B. Its authors fine-tuned it to read a text and a list of options and return a probability for each, in one forward pass. I run it with the same MLX pipeline and the Decision Index’s wording, and my runs match the index’s scores within a tenth of a point (95.7 vs. 95.8 on CLINC, 80.6 vs. 80.7 on VAST). No training on my side, and no few-shot (its format has no slot for examples).

OneJev reports its scores on the Decision Index, a community leaderboard of over 240 “decision models” that try to reproduce Jev, TypeSafe’s decision model. The index has two of my three tasks, CLINC with out-of-scope and VAST, and I score them exactly as it does. It gives me two reference points, drawn as dotted lines on the charts:

How I measured

#

• Splits: all three tasks use their official train and test splits.
• Quality: macro-F1 on the test set, so every class counts the same, as on the Decision Index.
• Cost: labeled examples used, training time, median latency per text at batch size 1, peak memory while predicting. All on the laptop, other GPU-heavy apps closed: TF-IDF on the CPU, Ettin on the GPU (PyTorch MPS), the 4B models on the GPU with MLX.
• One seed: each full training run and each few-shot draw happens once. Learning curves average three draws of the training examples up to 100 per class. Two numbers within a point are a tie, and on DynaSent, whose test set has only 720 sentences, within three.

Does the number of classes matter? CLINC150

#

CLINC150 is an intent classification dataset: short requests to a voice assistant, each filed under one of 150 intents spread over 10 domains (banking, travel, kitchen, work, and so on), plus an out-of-scope class for requests that match none of them.

Every intent has exactly 100 training and 30 test examples, which lets me cut the task down without changing anything else: I keep 10, 25, 50 or 100 intents and drop the others, and end with the full task, 150 intents plus out-of-scope. Which 10 intents you draw matters (ten banking intents is harder than ten from different domains), so at 10 and 25 I draw three sets and report the average.

For the prompts, intent names are made readable (pto_request becomes “pto request”), and few-shot gets one random example per intent: 151 on the full task, about 3,000 tokens. An intent keeps the same example in every slice, so the curve isn’t also a curve of different draws.

Macro-F1 on CLINC as intents are added. The last point is the full task, with the out-of-scope class. The two dotted lines are the Decision Index scores for Jev and the best entrant. The index has no slices, so compare them with the last point only:

More classes hurt the prompt, much less the encoder. With 10 intents, zero-shot Qwen gets 95.5 without any training example. At 100 intents it has lost 8.3 points, and it ends at 79.7 on the full task. Ettin-150M only loses 1.5 points from 10 to 100 intents. I see two reasons. The label list in the prompt gets longer (from about 30 to 470 tokens). And more intents means more neighbors that are hard to separate, like pto_request, pto_balance and pto_used. The choice of intents matters too: at 10 intents, zero-shot Qwen goes from 92.9 to 99.3 depending on the draw.

One random example per intent helps, but not on the full task. Few-shot Qwen is one to three points above zero-shot up to 100 intents: seeing one “pto balance” request helps to tell it from “pto used”. On the full task, the gain is gone (79.0 vs. 79.7, a tie). I think one random out-of-scope example can’t explain what “none of the above” means, and 151 examples (about 3,000 tokens) bury the label list. In any case, both prompts stay below TF-IDF at every size.

The decision model holds, like the encoder. OneJev1.6-4B is the same Qwen, trained to pick among options. It starts at 99.6, loses 1.7 points up to 100 intents, and ties the best trained model on every slice. On the full task it is ahead (95.7), and I gave it no labels. So the problem is not the LLM itself, it’s the generic prompt. One reservation: OneJev has seen some CLINC150 training examples (see the caveats).

TF-IDF is better than its reputation. Up to 25 intents it matches the encoder. From 50 intents it falls behind, because more intents share the same words. Intents are mostly about words: “time off” means PTO.

The out-of-scope class is where the full task gets hard. F1 on that class alone: 92.5 for OneJev, 78.1 for Ettin, 76.7 for zero-shot Qwen, 72.1 for few-shot Qwen and TF-IDF.

Or is it the kind of task? VAST

#

VAST is a stance detection dataset: given a New York Times comment and a topic, is the comment for, against, or neutral about that topic?

Only three readable labels. But most test topics never appear in training, and the stance is not in the words, it’s in how the comment relates to the topic: the first two rows are the same comment. That makes VAST the counterpoint to CLINC: few classes, hard task.

Few-shot gets five random examples per stance, 15 in total. By the logic of CLINC, three classes should be easy. Macro-F1 on the test set, with the same dotted lines as on the CLINC chart:

TF-IDF is at chance, and almost never predicts “neutral”. A bag of words sees which words are there, not if the comment talks about the topic.

The encoder reads the pair. After one hour of fine-tuning on the full training set, Ettin-150M gets 75.8, above Jev. “Neutral” even becomes its easiest class (88): once you compare comment and topic, an unrelated comment is easy to spot. The hard part is “for” vs. “against” on a topic never seen in training (68 and 72).

Prompting is in between, and random examples don’t help. Zero-shot Qwen does read the topic (47 on “neutral”) but is 22 points behind the encoder. With fifteen random examples, “neutral” improves (55), but “for” and “against” lose as much, and the total drops two points. None of the examples is about the test topic. OneJev is above everything (80.6), with no labels from me.

Latency is higher than on CLINC, as comments are longer: 15.6 ms per text for Ettin, 166 ms for OneJev, 195 ms for zero-shot Qwen.

When the words lie: DynaSent

#

DynaSent is a sentiment dataset: is a sentence about a restaurant or a shop positive, negative, or neutral? Easy, except for how it was built. In its second round, which I use, crowdworkers wrote sentences meant to fool a fine-tuned sentiment model (RoBERTa); about one in five did, and the dataset keeps every attempt, validated by five other workers:

The words are all there, they just point the wrong way. That’s the third case: not words that give the answer away (CLINC), not an answer hidden in a relation (VAST), but words that lie.

Few-shot gets 15 examples here too. The Decision Index has no DynaSent, so there are no dotted lines on the chart, and for OneJev I worded the request like the index’s VAST one. Macro-F1 on the test set:

TF-IDF picks up some signal, but not enough. Sentiment words still help a bag of words. But it can’t read “never not recommend” or “so good at being bad”, and it ends 17 to 20 points behind the others.

The four others are tied. They are within 2.7 points, and on this test set three points is noise. Zero-shot Qwen, with no labels and a two-line prompt, does as well as an encoder trained on all 13,065 sentences. OneJev loses the lead it had on CLINC and VAST, and only ties a plain prompt (note that I wrote its request for this task, not the index). My first guess was that sentences written to fool RoBERTa also fool Ettin, but only one in five fooled RoBERTa, so I’m not sure.

How many labels does the encoder need?

#

The prompts need no labels. The encoder does, but how many? The next chart shows how far Ettin-150M is from zero-shot Qwen as it gets more training examples per class. Zero-shot is the better of the two prompts on all three tasks, so above zero, the encoder beats both. Up to 100 per class, each point is the mean of three draws. The last point is the full training set (100 per intent on CLINC, about 4,400 per class on VAST and DynaSent):

On CLINC, 25 examples per intent are enough to beat both prompts. On VAST, the encoder needs 333 per class (about 1,000 examples). On DynaSent, it never gets above them, even with all the labels. Intents need a few examples, stance on unseen topics needs thousands, and with misleading words, all the labels only bring a tie.

What does each approach cost?

#

On the full CLINC task (150 intents + out-of-scope), quality against latency. The size of each bubble is the peak RAM while predicting:

The same numbers in a table (F1 is macro-F1, latency is the median per text, RAM is the peak while predicting):

And latency against the number of labels, from 3 (DynaSent) to 151 (full CLINC). At 10 and 25 intents, it’s the mean of three draws:

• The LLM pays for the label list on every prediction. At 151 labels, it takes 1.1 to 1.7 s per text, against 7.6 ms for the encoder. Zero-shot Qwen goes from 175 ms at 10 intents to 780 ms at 100. Few-shot is 55% slower on the full task: the examples are cached, but each label is scored against a 3,000-token prompt instead of 500. OneJev puts the options after the text, so they can’t be cached. With three labels, all three models answer in 100 to 150 ms.
• RAM is the real cost, at least the way I ran them. TF-IDF needs half a gigabyte, the encoder 1.5 GB. The 4B models peak around 18 GB on the full task (10 to 11 GB with three labels). That rules out most small servers, but quantization and a cheaper scoring would help (see the caveats).
• The LLM only saves labeling. It’s a real saving: 15,250 labels is days of work. How many you really need depends on the task, as the learning curves show.

So, do you need an LLM?

#

Well, it depends! Three tasks on one laptop are three data points, not a law. Your task, your labels and your hardware will change these numbers. So my best advice is to run the same kind of benchmark on your own data. A few hundred labeled examples for testing and an afternoon are enough to compare TF-IDF, a fine-tuned encoder and a prompt. It will tell you more than any rule of thumb, including this one. TF-IDF also tells you what kind of task you have. Close to the others: the words give the answer away. Near chance: the answer is in a relation. Above chance but far behind: the words may be misleading. With that said, here is how I would start:

flowchart TD
  A(["Do you have
labeled data?"])
  A -->|"No"| B(["Tight latency or
deployment constraints?"])
  A -->|"Yes"| C(["Tight latency or
deployment constraints?"])
  B -->|"No"| P1["Decision model
or LLM prompt"]
  B -->|"Yes"| D["Label with a large LLM,
distill into TF-IDF
or an encoder"]
  C -->|"Yes"| S["TF-IDF or a
fine-tuned encoder"]
  C -->|"No"| T(["What does
the task look like?"])
  T -->|"Words give
it away"| TF["TF-IDF, or an
encoder for a few
more points"]
  T -->|"Answer in
a relation"| E["Encoder with thousands
of labels, decision
model with fewer"]
  T -->|"Words
mislead"| P2["Encoder or LLM prompt:
a tie, the encoder
is cheaper to run"]

• No labels, no hard constraints: use something already trained. Try a decision model if one fits your task and your hardware: OneJev1.6-4B beat everything I trained on CLINC and VAST. But on DynaSent, the only task neither in its training data nor on the index, a plain zero-shot prompt did as well, so try both. A generic prompt gets worse with more classes, and a few random examples won’t fix it: they helped on small intent sets, but on none of the full tasks.
• No labels, but tight constraints: a 4B model is out, so you need labels from somewhere. The practical way is to let a larger LLM label a few thousand of your texts, then train a small model on them (a teacher and a student, or distillation). I didn’t test it, and the student will copy the teacher’s mistakes. But the learning curves give an idea of how much to label: a few dozen per class for intents, thousands for stance.
• Tight latency or deployment constraints: there are not many options. TF-IDF answers in under a millisecond on a CPU, with half a gigabyte. A ~150M encoder needs 7.6 ms (on the laptop’s GPU, more on a CPU) and 1.5 GB. The 4B models need 1 to 2 s and 18 GB.
• No hard constraints: it depends on the task.
  • The words give the answer away (intents, topics): start with TF-IDF, it’s a strong baseline up to a few dozen classes. A ~150M encoder adds 5.4 points on the full CLINC task.
  • The answer is in a relation (stance): an encoder, but only with thousands of labels. With the full VAST training set, Ettin-150M beats Jev and both prompts, in 16 ms per text instead of 195. OneJev is still better (80.6 vs. 75.8), but ten times slower. With only a few hundred labels, use a decision model.
  • The words mislead (sarcasm, negation, irony): either one. On DynaSent, 13,000 labels only gave the encoder a tie with a two-line prompt. If you have the labels, the encoder is cheaper to run. If not, use a prompt.

Caveats

#

As I said at the start, this is a weekend project. It was never meant to be paper-ready. Still, here are the limits I know of:

• Contamination. All three datasets have been public for years, and Qwen has likely seen them during pretraining. OneJev’s training data even includes 206 CLINC150 training examples (and BANKING77, another intent dataset). VAST and DynaSent are not in its listed sources, but it also uses samples of instruction collections that contain many classification datasets. This favors the Qwen-based models, especially OneJev on CLINC.
• Few-shot examples. Each few-shot number comes from one random draw. Another draw, or another order, could move VAST and DynaSent by several points. I also didn’t try picking the closest training examples for each text (retrieval), which usually works better.
• How I ran the LLMs. The BF16 weights take about 8 GB; 4-bit weights would take about 2.5 GB. But scoring each label separately also makes latency and memory grow with the number of labels (10 to 11 GB with three labels, 18 GB with 151), and quantization doesn’t change that part. Numbered labels read in a single token would matter more. Either way, I’d expect the encoder to stay far ahead.
• One seed, one prompt, no tuning. A better prompt or learning rate could move some numbers by a point or two. On DynaSent, the prompts and the OneJev request are mine, not the index’s.
• One LLM, not fine-tuned. A single 4B model stands for “an LLM” here. The best index entrants (27B and 31B) give an idea of what a bigger one brings: 97.6 on CLINC, 84.2 on VAST. Fine-tuning one with LoRA is for a later post.

What’s next

#

• How small can a classifier be? The encoder side: from 17M to 150M parameters, how many labels each size needs, CPU or GPU. Quantization too, for the encoders and the 4B models alike, so the cost comparison is fair. And how they compare with ready-made models that need no labels or only a few (sentence embeddings with a logistic regression, SetFit).
• Fine-tune or optimize the prompt? Same tasks: plain vs. GEPA-optimized prompt, base vs. LoRA weights.
• No labels, but an encoder anyway? A large LLM labels the training set, then a small encoder is fine-tuned on its answers (a teacher and a student). How close does the student get, how many labels does it need, and what does it cost?

Related reading

#

• Beating BERT? Small LLMs vs Fine-Tuned Encoders for Classification, by Alex Jacobs. Fine-tuned BERT and DeBERTa-v3 against small LLMs, zero-shot and few-shot, on SST-2, RTE, BoolQ and ANLI. DeBERTa wins three tasks out of four.
• Fine-Tuned “Small” LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification, by Martin Juan José Bucher and Marco Martini (2024). Fine-tuned BERT-style models against zero-shot GPT-3.5, GPT-4 and Claude Opus, stance included. The fine-tuned models always win. Same here, except on DynaSent, where it’s a tie.
• Do BERT-Like Bidirectional Models Still Perform Better on Text Classification in the Era of LLMs?, by Junyan Zhang et al. (Findings of EMNLP 2025). Encoders win on pattern-driven tasks, LLMs on tasks that need deep semantics. CLINC fits the first part. VAST only fits the second part for the decision model: the prompted LLM is 22 points behind the encoder. DynaSent fits better, with a tie.