Distilling LLM Decisions into Small Encoders

The question

This node asks how to transfer decisions and calibrated probabilities from large models into a small encoder. The question has two halves that are easy to conflate and important to separate. The first half is accuracy transfer: the student should pick the right label as often as the teacher. The second half is probability transfer: the student’s reported probabilities should mean what they say, so that a downstream automation gate can trust a 0.9 more than a 0.6.

The distinction matters because the standard distillation literature mostly optimizes the first half and says little about the second. A student can match its teacher’s argmax on nearly every example while reporting probabilities that are far more extreme than its hit rate justifies. For a typed decision model in the style of Jev, Laya, or the proposed Maestro Flash, that failure is load bearing: the product is the probability vector, not just the top label.

There is a further complication specific to this node. Most encoder distillation work distills from one bidirectional encoder into another, often within the same model family. Here the teacher is a large model, typically an autoregressive decoder-only large language model (LLM), and the student is a small bidirectional encoder. That cross-architecture gap closes off several transfer channels. Hidden states do not line up, attention maps have different shapes and different causal structure, and a teacher reachable only through an application programming interface (API) may expose no logits at all. What remains is response-based transfer (soft or hard targets over the decision space), data-side transfer (teacher-labeled or teacher-filtered training sets), and rationale-style auxiliary supervision, followed by a separate calibration refit on held-out data.

Background

Knowledge distillation, introduced by Hinton and coauthors, trains a small student to match the softened output distribution of a large teacher or ensemble rather than one-hot labels alone [1]. The soft targets carry what the literature calls dark knowledge: the relative probabilities the teacher assigns to wrong answers, which encode class similarity structure that hard labels discard. The teacher softmax is softened with a temperature, and the student matches it alongside ordinary supervised loss. The original paper reports compressing ensembles on MNIST and improving the acoustic model of a heavily used commercial system, plus a specialist-model variant for fine-grained classes [1].

Gou and coauthors frame distillation as model compression and acceleration for resource-limited devices, organizing methods into response-based, feature-based, and relation-based families [2]. Response-based methods match outputs. Feature-based methods match intermediate representations such as hidden states or attention maps. Relation-based methods match relationships between representations, such as pairwise similarities or attention relations, which relaxes the requirement that teacher and student share hidden sizes or head counts.

The encoder compression literature exercised all three channels on BERT-style teachers. DistilBERT distills during pretraining with a triple loss combining language modeling, distillation, and cosine-distance terms [3]. The Hugging Face documentation for the model restates the same triple objective [4]. TinyBERT introduced a two-stage framework with Transformer distillation at both the pretraining stage (general distillation from unfine-tuned BERT on general-domain text) and the task-specific stage (task-specific distillation from a fine-tuned teacher), matching embeddings, hidden states, and attention matrices layer by layer [5]. MiniLM proposed deep self-attention distillation: the student mimics the self-attention module of the teacher’s last Transformer layer, adding the scaled dot-product between values as new attention knowledge alongside the usual query-key attention distributions, and optionally routing through a teacher assistant of intermediate size [6]. MiniLMv2 generalized this to multi-head self-attention relation distillation over query, key, and value pairs, which removed the restriction that student and teacher share the same head count, and studied teacher layer selection instead of defaulting to the last layer [7].

A parallel line questioned how much of the recipe was needed. Turc and coauthors showed that simply pretraining compact models and fine-tuning them is competitive with more elaborate compression methods, then added standard task-knowledge distillation from large fine-tuned models on top of pretrained compact students, an algorithm they call pre-trained distillation, while studying the interaction of model size and unlabeled data properties [8]. The baseline discipline: a distilled student must beat a well-pretrained compact model fine-tuned directly, not just a randomly initialized one.

Calibration is a separate axis. Guo and coauthors found that modern neural networks, unlike those from a decade before, are poorly calibrated, identified depth, width, weight decay, and batch normalization as influencing factors, and reported that temperature scaling, a single-parameter variant of Platt scaling fit on held-out data, is surprisingly effective at calibrating predictions on most datasets [9]. Nothing in the standard distillation objective guarantees calibration transfer, so a distilled decision model generally needs its own temperature refit after training, on data the student did not train on.

What the evidence says

The headline results of the encoder distillation literature are strong on accuracy transfer within the encoder family. DistilBERT’s abstract reports a model 40% smaller than BERT that retains 97% of its language understanding capabilities while being 60% faster [3]. TinyBERT’s abstract reports a 4-layer student reaching more than 96.8% of its BERTBASE teacher’s GLUE performance while being 7.5x smaller and 9.4x faster at inference [5]; the project’s README repeats the 7.5x smaller and 9.4x faster figures and documents the general then task-specific two-stage procedure [10]. Adoption followed: the Hugging Face model API returned 7,946,165 downloads and 1,472 likes for distilbert-base-uncased at the time of this writing [11].

Multilingual compression shows the same pattern at larger scale. The MiniLM repository’s README reports two multilingual students distilled from XLM-R Large: a 12-layer 384-hidden model with 117M parameters, 2.7x speedup, XNLI accuracy 72.9, and MLQA F1 64.9; and a 6-layer 384-hidden model with 107M parameters, 5.3x speedup, XNLI 69.3, and MLQA 59.0 [12]. These repository-reported zero-shot figures show task-agnostic distillation preserving cross-lingual ability; they are not a controlled comparison.

Three results are directly relevant to the LLM-to-encoder direction. First, the step-by-step method of Hsieh and coauthors extracts rationales from an LLM and uses them as additional supervision in a multi-task framework, reporting that a fine-tuned 770M T5 outperforms a few-shot prompted 540B PaLM while needing less training data than either fine-tuning or plain distillation [13]. The supervision signal here is richer than a label but still textual, so it survives the API boundary that blocks hidden-state transfer. Second, MiniLLM addresses white-box LLM distillation by replacing the forward Kullback-Leibler (KL) divergence objective of standard distillation with reverse KL, which its authors argue is more suitable for generative language models because it keeps the student from overestimating low-probability regions of the teacher distribution [14]. Third, GLiNER demonstrates that a compact bidirectional encoder can take over a task previously associated with LLM prompting: its abstract reports a bidirectional transformer for open-vocabulary entity recognition with parallel extraction, outperforming ChatGPT and fine-tuned LLMs on zero-shot named entity recognition (NER) evaluations [15]. That result does not itself use LLM distillation, but it bounds what a well-trained small encoder can reach on a decision-shaped task.

The calibration evidence cuts the other way and is the most important part of this survey. Guo and coauthors report miscalibration as the default for modern networks and temperature scaling as the reliable fix [9]. The independent Jev benchmarks make the point concrete for decision models. The AbdelStark pilot reports Jev at Banking77/BTZSC accuracy 0.870 against GLiNER2.5 at 0.610, but on DAIR Emotion finds Jev substantially worse calibrated with Brier score 0.846 versus 0.668 [16]. The nibzard benchmark reports Jev at 76.3% on 77-way banking intent, behind gpt-oss-120b at 81.3% and glm-5.3 at 80.4%, with flat 264-276ms median (p50) latency from 2 to 255 options, cost $0.07 per 1,000 S1 decisions, and $28.34 in total measured provider spend [17]. Laya’s own BENCHMARKS.md, which warns that its Jev figures are third-party published and never measured in that repository, reports its headline comparison as typed-decisions 0.766 versus Jev 0.727, AG News 0.953 versus 0.910, DAIR Emotion 0.600 versus 0.480, expected calibration error (ECE) after temperature fitting 0.081 versus 0.246, and T4 p50 latency 32.8ms versus 236-276ms [18]. The recurring lesson is that accuracy and calibration move independently: a model can win on argmax while losing on Brier score or ECE, and distillation that only matches labels will happily transfer the argmax while saying nothing about the probabilities.

Method	Teacher to student	Transfer signal	Calibration handling	Relevance to LLM-to-encoder decisions
Hinton distillation [1]	Ensemble to single net	Soft targets with temperature	None in the objective	Defines the soft-target baseline; needs API or logits access
DistilBERT [3]	BERT to smaller BERT	Triple loss during pretraining	None reported	Task-agnostic encoder compression; same-architecture only
TinyBERT [5]	BERT to 4-layer BERT	Two-stage hidden and attention matching	None reported	Best same-family recipe; feature channels unavailable cross-architecture
MiniLM and MiniLMv2 [6][7]	Large to small Transformer	Self-attention and attention-relation distillation	None reported	Relation matching relaxes size constraints but still needs teacher internals
Pre-trained distillation [8]	Large fine-tuned to pretrained compact	Standard task distillation on pretrained student	None reported	Sets the baseline any distillation claim must beat
Distilling step-by-step [13]	`540B` PaLM to `770M` T5	Labels plus LLM rationales, multi-task	None reported	Textual signal survives API-only teachers; needs adaptation to single-pass heads
MiniLLM [14]	White-box LLM to small LM	Reverse KL on generations	None reported	KL-direction lesson for generative teachers; decision heads need their own divergence choice
Temperature scaling [9]	Any classifier to itself	Held-out single-parameter refit	The method itself	Required final stage; Laya extends it to per-bucket temperatures
GLiNER [15]	Supervised, no LLM teacher	Direct encoder training	Compared empirically, not transferred	Existence proof for small-encoder decision quality

Analysis

Four conclusions follow from the evidence, stated in decreasing order of confidence.

First, soft-target response distillation is the only transfer channel that survives every teacher constraint, and it is also the channel with the weakest calibration story. Feature and relation matching (TinyBERT, MiniLM, MiniLMv2) require teacher internals: hidden states, attention maps, or at least aligned layer structure [5][6][7]. A decoder-only LLM teacher breaks those assumptions structurally, and a closed API teacher removes them entirely. Soft targets need only a distribution over the decision space, which any teacher can in principle provide. But matching a teacher’s distribution is accuracy transfer with richer gradients, not calibration transfer. If the teacher is miscalibrated, and the available evidence says decision models often are [16][17], the student inherits the teacher’s confidence errors along with its knowledge. The fix used everywhere in the literature is a post-hoc refit on held-out data [9], which means the honest pipeline is distillation for accuracy, then refit for calibration, with the refit evaluated on data neither stage has seen.

Second, the teacher’s temperature and the student’s temperature are different objects and must be fit separately. The teacher temperature controls how much dark knowledge the soft targets reveal. The student temperature controls how the student’s logits map to reported probabilities after training. Collapsing them into one value, or shipping the teacher’s temperature as the student’s calibration, has no justification in any source reviewed here. Laya’s practice of fitting one temperature per question-type and option-count bucket, with clamping to [0.5, 5.0] as a defensive bound, is the closest decision-model precedent for the student side [18]. The teacher side has no decision-model precedent in the reviewed sources: the temperature or sampling scheme that produces the best soft targets for a choice head over 77 options is UNVERIFIED and needs a direct ablation.

Third, rationale supervision is promising but structurally mismatched to single-pass encoders, and the mismatch defines the engineering problem. Step-by-step distillation works because the student is itself a sequence model that can be trained to emit rationales as an auxiliary task [13]. A typed-decision encoder with choice, score, and boolean heads emits no tokens, so there is nowhere for a rationale to go at inference time. The viable adaptations are all indirect: use rationales to filter or weight training data (keep examples where the teacher’s rationale supports its label), use rationale consistency as a per-example loss weight, or train an auxiliary head that predicts rationale-derived features and is discarded at inference. Whether any of these beats plain soft-target distillation on decision accuracy at matched data budgets is UNVERIFIED. The step-by-step result (a 770M student beating a 540B few-shot teacher [13]) motivates the experiment without settling it, because the student there is generative.

Fourth, the KL direction question raised by MiniLLM has a decision-head analogue that the literature does not answer. MiniLLM replaces forward KL with reverse KL for generative students to avoid overestimating low-probability regions [14]. For a choice head, forward KL (teacher as reference) is mode-covering: the student spreads mass everywhere the teacher puts mass, which preserves dark knowledge but can flatten the top probability. Reverse KL (student as reference) is mode-seeking: the student concentrates on the teacher’s peaks, which sharpens decisions but discards the tail structure that makes soft targets informative. Which direction produces better Brier score and ECE after a held-out refit, at matched accuracy, is UNVERIFIED for decision heads. The same question applies to score heads with ordinal structure, where neither plain KL direction respects ordering and a ranked-probability-style objective may be the right distillation loss. No reviewed source tests this.

A secondary point concerns sequencing with later training stages. The Maestro Flash proposal stages training as supervised fine-tuning, then soft-distribution distillation from Qwen3 or Gemma 3, then proper-scoring-rule reinforcement learning (RL) with a baseline in the style of Group Relative Policy Optimization (GRPO), then held-out temperature fitting. The GRPO algorithm was introduced in the DeepSeekMath report, whose abstract credits it alongside data selection for 51.7% on MATH without external toolkits [19]. Sequencing distillation before RL is coherent: distillation transfers the teacher’s decision boundaries, RL against proper scoring rules then reshapes the student’s own probabilities toward calibration, and the final temperature refit corrects whatever miscalibration remains. But each stage can undo the previous one’s calibration work, so the pipeline needs calibration measured after every stage, not just at the end. A distillation stage that improves accuracy while doubling ECE, followed by an RL stage evaluated only on reward, would look like progress while shipping worse probabilities.

Implications for small decision models

For Jev, the implication is that behavioral distillation is possible but calibration-limited. Jev is a closed API, so only response-based transfer is available: query it over a large unlabeled set, collect its decisions and probabilities, and train an encoder student on them. The nibzard benchmark’s cost figure ($0.07 per 1,000 S1 decisions [17]) makes the data-collection budget computable in advance, which is genuinely useful for planning. But the same benchmark suite reports Jev’s ECE at 0.246 against LLM competitors ranging far lower (as summarized in the Jev survey on this site), and the AbdelStark pilot finds Jev losing on Brier score where it wins on accuracy [16]. A student distilled from Jev responses would inherit that miscalibration unless the pipeline includes a refit on labeled held-out data, and the refit needs ground truth, not more teacher outputs. There is also a licensing and terms-of-service question around distilling a commercial API that this survey does not resolve; it is flagged, not answered.

For Laya, the implication runs in the opposite direction: Laya is open, so every transfer channel is available, but its known failure modes would transfer with its knowledge. Laya’s Apache-2.0 checkpoints expose hidden states and attention maps, which means TinyBERT-style or MiniLM-style feature distillation from Laya into a smaller encoder is technically straightforward. The problem is architectural. The Laya teardown on this site documents the option-token budget failure (options share a fixed head_max_len with instructions, so Banking77 labels get roughly 3-4 tokens each) and Laya’s own BENCHMARKS.md confirms weak high-cardinality behavior alongside strong low-cardinality results [18]. Distilling Laya’s decisions into a student with the same joint option encoding preserves the failure. Distilling into a student with decoupled option encoding changes the architecture enough that feature matching no longer applies, leaving response-based transfer. Either way, the distillation study must report high-cardinality accuracy separately, or it will average a preserved failure into an acceptable-looking mean. Laya’s per-bucket temperature practice is the part most worth imitating directly [18].

For the proposed Maestro Flash, which has no weights and no results, the implication is a concrete experimental ordering for its distillation stage. The proposal plans soft-distribution distillation from Qwen3 or Gemma 3 between supervised fine-tuning and proper-scoring-rule RL. The evidence reviewed here suggests four design choices. One, use response-based distillation with teacher temperatures ablated, since feature channels are unavailable from decoder teachers and the right teacher temperature for decision heads is UNVERIFIED. Two, run the Turc baseline first: a well-pretrained compact encoder fine-tuned directly, which distillation must beat before any feature-matching machinery is justified [8]. Three, test one rationale-derived signal (filtering, weighting, or an auxiliary head) against plain soft targets at a matched data budget, since the step-by-step result motivates but does not settle the question for non-generative students [13]. Four, measure Brier score and ECE after every stage on data held out from all stages, because distillation, RL, and temperature fitting each move calibration independently and the proposal’s go/no-go criteria depend on knowing which stage moved it. This is the harness the proposal’s first exit condition already demands: one that refuses to report accuracy without calibration.

Limitations

This survey read abstracts, repository READMEs, model APIs, and official documentation, not full paper texts or codebases, and it ran no experiments. Every quantitative claim is attributed inline to the source that reported it, and those figures are self-reported by their authors under their own protocols, not independently reproduced here. Cross-paper comparisons (for example, DistilBERT’s 97% against TinyBERT’s 96.8% [3][5]) are not like-for-like: the teachers, tasks, and metrics differ, and the percentages should be read as each paper’s own summary of its own result.

Three specific gaps limit what this survey can conclude. First, no reviewed source measures calibration transfer directly: whether a student distilled from a calibrated teacher stays calibrated, or how much held-out data the refit needs as a function of option count. Guo and coauthors study calibration of directly trained networks [9], and the decision benchmarks measure calibration of deployed models [16][17][18], but the distillation step between them is unmeasured. Second, the LLM-to-encoder direction has thin direct evidence. Step-by-step distillation uses a generative student [13], MiniLLM distills into smaller generative models [14], and GLiNER trains its encoder without an LLM teacher [15]. The exact configuration this node asks about (decoder LLM teacher, encoder student, decision heads, calibrated outputs) appears in no reviewed source as a reported experiment. Third, costs are incompletely observed. Teacher-API pricing is public and the nibzard benchmark’s $28.34 total spend is measured [17], but graphics processing unit (GPU) training costs for the distillation runs themselves are planning estimates in the proposal, not measurements, and should stay labeled that way.

Open questions

What teacher temperature and KL direction for soft-target distillation produce the best post-refit Brier score and ECE on high-cardinality choice heads at matched accuracy?
Does rationale-derived supervision (filtering, loss weighting, or an auxiliary head) beat plain soft-target distillation for single-pass encoder decision heads at a matched teacher-query budget?
How much labeled held-out data does the calibration refit need as option count grows, and does per-bucket temperature fitting stay sufficient or does it require per-language or per-domain grouping?

References

← All papers