In the previous post, the two ladders (decision models and fine-tuned encoders) left a gap in the middle. A decision model needs no labels, but it is slow: on CLINC, OneJev1.6-4B took two seconds per text. A fine-tuned Ettin answers CLINC’s short queries in 1 to 9 ms, but needs hundreds of labels per class. You may not have them, and you may be weighing it up: spend hours annotating data, or just use a decision model?
Distillation promises both. Let a decision model (the teacher) label your texts once. Then fine-tune a small encoder (the student) on its answers, as if they were real labels. No labels from you, and the serving cost of a fine-tuned encoder. And since the teacher runs only once, it can be big and rented, a few cents away behind an API. The catch: the student learns from a teacher that makes mistakes. So how much of the teacher survives the trip, and what do its mistakes become?
Two words for the rest of the post. A gold label is the real answer, given by a person: here, it is only used to grade. A silver label is the teacher’s answer: free, plentiful, and sometimes wrong. The student only ever sees silver.
The teacher is slow but runs once. The student is fast, and is the one you serve. Gold labels are only used to grade, at the very end.
Four questions:
- How close does the student get? To its teacher, and to the same encoder trained on real labels.
- Hard or soft labels? Does the student learn more from the teacher’s full probabilities than from its top answer?
- Does a better teacher make a better student? From 2B to 27B, in two families of decision models. And can the student beat its teacher?
- What does it cost? Renting a teacher once and training, against serving the teacher forever.
The students train on the same laptop (Apple M5, 24 GB of unified memory), with the same code: driquet/local-classification, in experiments/distillation. Same warning: a weekend project, one task, three seeds where the numbers are close and one elsewhere. It gives the gist, not a law.
TL;DR · in a hurry?
Without a single label, the student scores within a point of its teacher, mistakes included. So the best teacher makes the best student: here Decider 27B, which labeled 9,000 papers for 8 cents. Its student answers in 32 ms on the laptop's GPU, with no API, but stays 5 points below the same encoder trained on real labels.
A new task: which field is this paper from?#
The first two posts used CLINC, VAST and DynaSent: classic datasets, the kind decision models are trained on (tasksource alone gathers about 750). A teacher that has already seen the gold labels would make distillation look better than it is. So I wanted a task the teachers are unlikely to have trained on.
arXiv papers from 2026. Every paper on arXiv has a primary category, picked by its authors when they submit it. The task: read the title and abstract, and guess the category. I kept eight computer science categories, and only papers first submitted between March and October 2026, from the arXiv metadata snapshot.
The teachers themselves came out later, so in theory they could have seen these papers. I consider the risk low. Their base model, Qwen3.5 for all but one, came out at the end of February, before the first of these papers. Their fine-tuning is about decisions in general, not this year’s arXiv. The one teacher on a newer base, Decider 27B on Qwen3.8 (August 2026), could have seen the papers from March to July. Yet it scores level with Mammoth 27B on Qwen3.5: a teacher that had read the answers would stand out. Its mistakes don’t look like recall either: it files papers by their method, where the authors file them by their field (more on that below).
| Category | Name given to the teachers | Example (title) |
|---|---|---|
| cs.CL | Computation and Language (NLP) | “Tracing Pharmacological Knowledge In Large Language Models” |
| cs.CV | Computer Vision | “NanoGS: Training-Free Gaussian Splat Simplification” |
| cs.LG | Machine Learning | “Large Spikes in Stochastic Gradient Descent: A Large-Deviations View” |
| cs.CR | Cryptography and Security | “Unclonable Encryption in the Haar Random Oracle Model” |
| cs.RO | Robotics | “Multifingered force-aware control for humanoid robots” |
| cs.SE | Software Engineering | “Coverage-Guided Multi-Agent Harness Generation for Java Library Fuzzing” |
| cs.HC | Human-Computer Interaction | “Understanding the Effects of Interaction on Emotional Experiences in VR” |
| cs.IR | Information Retrieval | “Negative Sampling Techniques in Information Retrieval: A Survey” |
Easy to explain, easy for a bag of words with labels, and harder for a model without them.
Gold labels, only for grading#
1,000 papers per category form the training pool (8,000 papers). Their gold labels are set aside before the teacher sees anything. The validation set (125 per category) is labeled by the teacher too: no gold label touches training. The test set (250 per category, 2,000 papers) keeps its gold labels, and is only read at the very end.
The gold labels of the training pool have two uses only: training the ceiling (the same encoder on real labels), and measuring how often the teacher is wrong on the data the student learns from.
The contestants#
Teachers · no labels
Base Labs Joey · Elephant · Mammoth
Perplexity Decider Flash · Decider
OneJev1.6-4B
Decision models from 2B to 27B, among them OneJev1.6-4B from part 1. They label the training pool.
Student · the teacher's labels
Ettin-150M
Fine-tuned on a teacher's answers, on the laptop. One student per teacher.
Ceiling · gold labels
Ettin-150M
The same encoder, fine-tuned on the real labels: what labeling by hand would buy.
Floor · both
TF-IDF + logistic regression
On the gold labels, and on the teachers' labels: does the student need to be an encoder?
Two families, rented#
Why rent the teachers? The teacher runs only once, so it doesn’t have to live on the laptop. And renting is simpler (one API for every model), much faster than running them locally, and cheap: $0.86 for every label in this post. Besides, the 27B teachers are too big to serve on the laptop.
OpenRouter serves decision models behind one API: you send a text, a question and the options, and get back a probability per option. Nothing is generated. Each provider runs its model with its own prompt template, in its own best setup, which is exactly what I want from a teacher. I picked the two families that come in several sizes and rank high on the Decision Index, the public leaderboard of decision models:
| Teacher | Maker | Size | Base | Decision Index | Price, per M input tokens |
|---|---|---|---|---|---|
| Joey 1 | Base Labs (Baseten) | 2B | Qwen3.5-2B | 35.9 | $0.010 |
| Elephant 1 | Base Labs (Baseten) | 9B | Qwen3.5-9B | 60.3 | $0.035 |
| Mammoth 1 | Base Labs (Baseten) | 27B | Qwen3.5-27B | 68.4 | $0.100 |
| Decider Flash v1.2 | Perplexity | 4B | Qwen3.5-4B | 54.3 | $0.010 |
| Decider v1.2 | Perplexity | 27B | Qwen3.8-27B | 68.0 | $0.020 |
| OneJev1.6-4B | OmniJev | 4B | Qwen3.5-4B | 48.9 | on the laptop |
Decision Index 0.4.1 (November 2026), full score. A paper with its question and options is about 430 tokens with Perplexity's prompt template, 445 with Base Labs'.
OneJev1.6-4B stays as the local option: the teacher you can run when your texts can’t leave the machine.
The teachers all get the same prompt: the question “Which arXiv category is this paper’s primary one?”, the title and abstract, and the eight names above as options A to H (the Decision Index layout). Each returns a probability per option. Their top answer is the hard label; the full distribution is the soft label. For example, here is what OneJev1.6-4B says about “Compact convolutional neural networks for AI-based drone detection system”, filed by its authors under cs.LG:
One paper, as the teacher sees it. The hard label is its top answer, cs.CV: wrong. The soft label keeps the whole distribution, and the gold answer, cs.LG, is a close second. On this paper, the student trained on these soft labels answers cs.LG.
The student is the Ettin-150M of part 2, with its hyperparameters (learning rate 5e-5, 4 epochs, texts cut at 512 tokens). Only the targets change. No checkpoint selection: the student is the one at the end of training. The validation set only tracks the loss during training.
How I measured#
- Quality: macro-F1 on the 2,000 gold test papers, so every category counts the same, with a 95% bootstrap interval (about ±2 points). Where results are close, I ran two or three seeds and report the mean; elsewhere, one seed: it’s a weekend project. Seeds move a student by up to 1.3 points, so a gap under a point between two single-seed students means nothing. Two models graded on the same test papers can be compared more finely than ±2, but I only lean on gaps that hold across seeds.
- Agreement: how often the student gives the same answer as its teacher, right or wrong.
- Hard labels: cross-entropy on the teacher’s top answer.
- Soft labels: KL divergence to the teacher’s probabilities, softened with a temperature of 2 (Hinton et al., 2015).
- Cost: labeling time and price for the pool and validation (9,000 papers), training time, median latency per paper at batch size 1, peak RAM while predicting.
How close does the student get?#
Macro-F1 on the gold test set, for each teacher and for its student (soft labels). The dotted lines: the same encoder trained on the 8,000 gold labels, and TF-IDF trained on them.
Means over seeds where I ran several. Joey 1 (48.7) is left out: too weak to be worth a student, see below.
Every student lands on its teacher. 79.4 against 79.8 for Decider 27B, 78.9 against 79.3 for Mammoth 27B, 77.3 against 76.5 for Elephant 9B: five teachers, and no student more than 0.8 points away, either way. Within the noise. The student never sees a gold label, and it ends up as right as the model that taught it, within the noise. Never clearly better, never clearly worse.
The gold labels are 5 points away. The best students (79.4 and 78.9) stay below Ettin trained on real labels (84.7). And TF-IDF trained on those same real labels gets 83.9: on arXiv, the words give the answer away, as on CLINC in part 1. A bag of words is enough, if its labels are right. The bottleneck is the teacher, not the student.
With the teacher’s labels, the encoder does earn its keep. On every teacher’s labels, TF-IDF trails the Ettin student trained on the same labels: by 3.0 points on Decider’s (76.4), 3.2 on Mammoth’s (75.7), 3.3 on OneJev’s (71.3), 4.6 on Elephant’s (72.7) and on Decider Flash’s (67.2). On gold labels the two are level; on silver ones, the encoder copes better with the noise.
Where it goes wrong: cs.IR, cs.CR, cs.CL and cs.LG. Per category, for the best teacher:
| Category | Decider 27B | Its student | Ettin-150M, gold |
|---|---|---|---|
| cs.CL | 67 | 69 | 76 |
| cs.CV | 85 | 85 | 86 |
| cs.LG | 69 | 67 | 75 |
| cs.CR | 79 | 80 | 87 |
| cs.RO | 92 | 91 | 91 |
| cs.SE | 83 | 83 | 87 |
| cs.HC | 85 | 84 | 88 |
| cs.IR | 77 | 77 | 88 |
| Macro-F1 | 79.8 | 79.4 | 84.7 |
F1 per category on the gold test set. Teacher and student are colored by their gap to the encoder trained on gold labels, in grey: within 1 point 1 to 3 behind 3 to 6 over 6.
The teacher files papers by their method, the authors by their field. Of the 250 retrieval papers, Decider 27B puts 58 under NLP and 24 under Machine Learning; of the 250 security papers, 35 under Machine Learning and 23 under NLP. A recommender built on an LLM is about an LLM: reasonable, but not what the authors chose. cs.CL and cs.LG pay the other side: the misfiled papers land there, which lowers their precision. The bigger teachers make this mistake less often: OneJev1.6-4B put 54 security papers under Machine Learning and 41 under NLP, and Mammoth 27B only 18 and 9 (F1 88 on cs.CR, level with the gold encoder). Where the name of the category leaves little doubt (Robotics, Computer Vision, HCI), the teacher is within about 3 points of the gold encoder. The student keeps the same profile, category by category.
Does the student copy its teacher’s mistakes?#
On the test set, Decider 27B is wrong on 418 papers. What does its student (seed 0) answer on those?
Same mistake
67%
The student copies the teacher's wrong answer.
Fixed
27%
The student gives the gold label.
Another mistake
6%
Wrong too, but differently.
Most mistakes carry over, and they are the systematic ones: retrieval and security papers filed by their method. The student fixes 112 of the teacher’s mistakes, and makes 111 new ones on papers the teacher got right: the two cancel out. The better the teacher, the more of its mistakes the student fixes (27% for Decider 27B, 18% for Decider Flash), but the new mistakes keep pace. Overall, the student agrees with its teacher on 87.5% of the test papers (88 to 90% for the other pairs). It learns the teacher’s idea of the categories, bias included.
Takeaway
Without a single label, the student matches its teacher, mistakes included: the teacher sets the ceiling.
Hard or soft labels?#
The same students, trained on the teacher’s top answer (hard) or on its probabilities (soft), for the local teacher and the two 27B rented ones:
Means over seeds where I ran several.
Soft labels help two teachers out of three. With OneJev1.6-4B, soft beats hard by 1.6 points (74.6 against 73.0), on both seeds. With Mammoth 27B, by 1.0 (78.9 against 77.9), and every soft seed beats every hard one (78.3 to 79.6 against 77.5 to 78.2). With Decider 27B, the two are level: 79.4 soft, 79.5 hard, and the seeds overlap (soft: 78.9 to 79.9; hard: 79.1 to 80.3).
And it’s not about how confident the teacher is. I expected a hesitant teacher to put more in its probabilities. OneJev1.6-4B’s top answer gets 0.66 on average, Decider 27B’s 0.73, Mammoth 27B’s 0.80: the most confident teacher is the one where soft labels help, and the one in the middle is where they don’t. Three teachers is too few to tell what makes the difference.
Soft labels cost nothing more: the teacher returns the probabilities anyway, and training takes the same time. If your teacher gives probabilities, use them.
Takeaway
Keep the teacher's probabilities: they cost nothing and never hurt.
Does a better teacher make a better student?#
Each dot is one teacher and its student. Left to right: how good the teacher is. Bottom to top: how good its student is. On the dashed line, the student scores exactly like its teacher.
One dot per teacher, labeled with the teacher's name; its height is the score of the Ettin-150M student trained on its soft labels. Means over seeds where I ran several.
Yes, point for point. Every dot sits on the line: a 7.4-point gap between teachers (Decider Flash to Decider) becomes a 7.6-point gap between students. Pick the best teacher you can afford to run once.
Bigger helps, less and less. Within a family, the teachers line up by size:
| Family | 2B | 4B | 9B | 27B |
|---|---|---|---|---|
| Base Labs | Joey: 48.7 | Elephant: 76.5 | Mammoth: 79.3 | |
| Perplexity | Decider Flash: 72.4 | Decider: 79.8 |
Teacher macro-F1 on the gold test set.
The big step is from 2B to 9B: Joey 1 files half the papers under Machine Learning, and a teacher this weak isn’t worth a student. From 9B to 27B, the Decision Index gains 8 points; arXiv only 2.8. The two 27B teachers end up level (79.3 and 79.8, within the noise), still 5 points under the gold encoder. What’s left is mostly about the authors’ conventions, which no teacher can read off an abstract.
The leaderboard doesn’t pick the teacher for you. OneJev1.6-4B (48.9) has the lowest Decision Index score of the teachers, after Joey, and it beats Decider Flash (54.3) by 2.7 points here. The index averages dozens of tasks; yours is one. A pilot on 100 papers per split, for 2 cents across all five rented teachers, got the order right except for two teachers 1.4 points apart. Run one before you pick.
Silver labels, measured. On the training pool, the teachers’ top answer matches the gold label 72% (Decider Flash), 74% (OneJev1.6-4B), 75% (Elephant) and 78% of the time (Mammoth and Decider). Even with the best teacher, more than a fifth of the labels are wrong, and the student learns them as they are.
Takeaway
Spend your money on the teacher, once, after a pilot on your own data.
What does it cost?#
On the arXiv task. Labeling is the training pool plus validation (9,000 papers): through the API for the rented teachers, one request at a time, and in bf16 with MLX on the GPU for OneJev1.6-4B. Serving is on the laptop’s GPU: fp32 with PyTorch (MPS) for the encoders, Q4_0 with llama.cpp for OneJev. TF-IDF runs on one CPU thread, and the rented teachers through the API. Serving on a CPU, with int8 encoders, was part 2’s topic.
| Your labels | F1 | Labeling, once | Training | Latency | RAM | Per million papers | |
|---|---|---|---|---|---|---|---|
| Decider 27B, via the API | 0 | 79.8 | none | none | 316 ms | none | $8.60 |
| Mammoth 27B, via the API | 0 | 79.3 | none | none | 268 ms | none | $44.50 |
| OneJev1.6-4B, on the laptop | 0 | 75.1 | none | none | 456 ms | 4.1 GB | GPU time |
| Student of Decider 27B | 0 | 79.4 | 60 min, $0.08 | 1.3 h | 32 ms | 1.7 GB | GPU time |
| Student of OneJev1.6-4B | 0 | 74.6 | 49 min | 1.3 h | 32 ms | 1.7 GB | GPU time |
| Ettin-150M, gold labels | 8,000 | 84.7 | 65 to 130 h by hand | 1.3 h | 32 ms | 1.7 GB | GPU time |
| TF-IDF, gold labels | 8,000 | 83.9 | 65 to 130 h by hand | 33 s | 1.1 ms | 0.5 GB | one CPU thread |
teachers students gold labels. F1 on the 2,000 test papers. Latency: median per paper at batch size 1, as served; through the API, it includes the round trip from the laptop. Labeling the five rented teachers, test sets included, cost $0.86. Labeling by hand is an estimate: 30 seconds to a minute per abstract, for someone who knows the field.
The money is not the point. Labeling 9,000 papers with the best teacher costs 8 cents, and serving it costs $8.60 per million papers: a student won’t pay for itself in dollars. Mammoth, at the same quality, costs five times more, which the leaderboard won’t tell you either.
The point is everything else. The student answers in 32 ms on the laptop’s GPU, ten times faster than the API. No network, no rate limit, no API key, and no provider who can change or retire the model under you. Your texts leave the machine once, to be labeled, and never again. And if they can’t leave it at all, OneJev1.6-4B labels them locally, in under an hour, for 4.8 points less.
What the gold labels would buy. 5.3 points over the best student, and they make the student itself optional: TF-IDF on the same labels trains in 33 seconds and answers in a millisecond. But 8,000 labels by hand are 65 to 130 hours, two to three weeks of full-time work, against an hour of API calls; part 1 suggests a few hundred per class would already get close on a task where the words give the answer away. That’s for a later post.
So, should you distill?#
Back to the starting point: you have a classification task, but no gold labels. You can call a decision model on every text, or train a small model on its answers. Or label the data yourself. Here is how I would decide, with the numbers from arXiv:
Numbers from this post: macro-F1 on the arXiv test set, latency per paper as served.
Two steps cost a little time but save a lot. The pilot: label 100 of your texts by hand, about an hour, and grade the candidate teachers on them (2 cents for all five here). And whichever branch you take, check a few hundred of the teacher’s labels by hand: what it gets wrong every time, the student will too.
This is also what we did in the paper#
The paper
Distilling Human-Aligned Privacy Sensitivity Assessment from Large Language Models
Gabriel Loiseau, Damien Sileo, Damien Riquet, Maxime Meyer, Marc Tommasi (LEGAL / CALD-pseudo workshop, 2026). Models, code and data on GitHub.
Same recipe, bigger scale, on another question: how private is a text?, from 1 (harmless) to 5 (extremely private), on the scale of Meisenbacher et al. (2025). Their study showed that a large LLM agrees with people on this about as well as people agree with each other. But sending private texts to an API to ask how private they are defeats the purpose: the evaluator has to be small, and local. So Mistral Large 3 (675B) rated 200,000 user-written texts, Ettin students learned its ratings, and 250 texts rated by 677 people served as gold, with agreement measured by Krippendorff’s α (1: perfect, 0: chance).
| Model | Parameters | Agreement with the average human rating (α) |
|---|---|---|
| Mistral-7B, prompted | 7B | 0.563 |
| Mistral Large 3, the teacher | 675B | 0.716 |
| Ettin-17M, its student | 17M | 0.708 |
| Ettin-150M, its student | 150M | 0.737 |
On the 250 texts rated by people, from the paper's Table 5.
There, the student beat its teacher, with 4,500 times fewer parameters. We read it as denoising: an LLM’s answers vary from one prompt to the next, and 200,000 of them smooth out into one stable decision boundary. That didn’t happen here: the teachers’ mistakes on arXiv are systematic, not scattered, and 8,000 examples are not 200,000. The warning holds at both scales: the student inherits the teacher’s biases, which is why both measure against people.
Caveats#
- One task. arXiv categories are a task where the words give the answer away. On a task where the answer is in a relation, like VAST, the teacher’s mistakes may be harder to average out.
- Gold labels are the authors’ choice. cs.LG vs. cs.CV is a matter of opinion, and part of the gap to the ceiling is in the labels, not in the models.
- Contamination. The teachers came out after these papers, and an arXiv page shows the primary category right next to the abstract: a base model trained on a later crawl could have read both. The risk is low, not zero. Decider 27B is built on Qwen3.8, released in August 2026: papers from March to July could be in its training data. It still scores level with Mammoth 27B, on Qwen3.5.
- Rented teachers move. The API models are named by version (Decider v1.2), but the provider runs them, and can update or retire them. The labels are saved with the code; the teachers may not be there in a year.
- One student size, one temperature. Ettin-150M only, soft labels at a temperature of 2, no tuning, no checkpoint selection.
What’s next#
- A few gold labels on top? Fine-tune the student, or a small teacher, on 10 to 100 gold labels per class after distillation.
- Fine-tune or optimize the prompt? Plain vs. GEPA-optimized prompt, base vs. LoRA weights.
Related reading#
- Distilling the Knowledge in a Neural Network, by Geoffrey Hinton, Oriol Vinyals and Jeff Dean (2015). Soft labels and temperature: where the idea of learning from the teacher’s probabilities comes from.
- Want To Reduce Labeling Cost? GPT-3 Can Help, by Shuohang Wang et al. (Findings of EMNLP 2021). An LLM labels, a small model learns: the same recipe, with GPT-3 as the teacher.
- Distilling Step-by-Step!, by Cheng-Yu Hsieh et al. (Findings of ACL 2023). Small students beating much larger teachers, with fewer labels.

