Teacher Temperature and KL Direction

This paper is a child of the distillation survey, which left the question of teacher temperature and Kullback-Leibler (KL) divergence direction UNVERIFIED. It builds on that survey without repeating its method tables or benchmark summaries. All arXiv identifiers (IDs) below were verified against their arXiv abstract pages, and every number is attributed inline to a source fetched for this paper. Velofy has trained no model for this paper and ships none with it.

The question

What teacher temperature and KL direction for soft-target distillation produce the best post-refit Brier score and expected calibration error (ECE) on high-cardinality choice heads at matched accuracy?

The question names two knobs, one metric pair with a qualifier, and one setting. The knobs are the teacher temperature T_t, which softens the teacher distribution that becomes the soft target, and the KL direction of the distillation loss: forward KL with the teacher as reference, reverse KL with the student as reference, or a symmetric combination. The metric pair is Brier score and ECE measured after a held-out calibration refit, because the distillation survey on this site establishes that the teacher temperature and the student temperature are different objects that must be fit separately. The qualifier is matched accuracy: the scikit-learn calibration documentation warns, following the Brier score decomposition of Murphy, that a lower Brier loss does not necessarily mean a better calibrated model, since the score mixes calibration with discriminative power [15]. Any comparison of distillation settings on Brier score is confounded unless accuracy is held fixed. The setting is high-cardinality choice heads, on the order of 77 options, where most soft-target mass sits on non-target classes and tail behavior dominates the loss.

Background

Knowledge distillation (KD) trains a student to match the softened output distribution of a teacher alongside ordinary supervised loss. Hinton and coauthors introduced the temperature parameter that softens the teacher softmax, so that the relative probabilities of wrong answers (dark knowledge about class similarity) survive in the training signal [1]. In the standard formulation the matching loss is forward KL: the PyTorch KLDivLoss documentation defines the pointwise loss as target * (target.log() - input), with the teacher distribution as target and the student output as input in log-space, and warns that only reduction="batchmean" returns the mathematically correct KL value [14]. Forward KL is mode-covering: the teacher distribution weights the loss, so the student is penalized wherever it assigns too little mass where the teacher puts mass, and it spreads mass to cover the teacher’s support. Reverse KL swaps the roles, weighting by the student distribution, which is mode-seeking: the student concentrates on the teacher’s peaks and may ignore low-probability regions.

From first principles, temperature changes only spread, and each extreme loses information. Dividing the logits by a temperature keeps the class order and changes only how spread out the probabilities are. As the temperature approaches zero, the soft target becomes a one-hot vector on the teacher’s top class and says nothing about the wrong answers. As it grows very large, the target flattens toward uniform, and the gaps that show which wrong answers are close shrink toward nothing. A useful teacher temperature sits between those ends. The student refit is one more scalar of the same kind, applied last: it can fix overall sharpness, but it cannot restore class orderings the student never learned. Those two facts frame every conclusion below.

Calibration is the separate axis this node cares about. Guo and coauthors found that modern networks are poorly calibrated and reported that temperature scaling, a single-parameter refit on held-out data, is surprisingly effective on most datasets [2]. ECE measures the gap between predicted probabilities and empirical accuracies across confidence bins. Brier score is the mean squared error between predicted probabilities and one-hot outcomes, and its decomposition into calibration, resolution, and uncertainty terms is why the matched-accuracy qualifier is load bearing [15].

Two reformulations of distillation shape everything that follows. Yuan and coauthors prove that knowledge distillation is a type of learned label smoothing regularization, and report that reversing the procedure (student teaching teacher) still helps and that a poorly trained, low-accuracy teacher can still improve the student significantly [4]. Zhao and coauthors split the classical KD loss into target-class knowledge distillation (TCKD), which transfers sample difficulty, and non-target-class knowledge distillation (NCKD), which they identify as the prominent reason logit distillation works; the classical coupled formulation suppresses NCKD, so they propose Decoupled Knowledge Distillation (DKD) to weight the two parts independently, reaching results comparable to or better than complex feature-based methods on CIFAR-100, ImageNet, and MS-COCO [5]. For a 77-way head the DKD split matters more than for a 10-way head: with so many non-target classes, NCKD carries most of the dark knowledge, and anything that suppresses it (including a poorly chosen teacher temperature) removes most of the point of distillation.

What the evidence says

The evidence falls into three groups: teacher temperature, teacher calibration, and KL direction. No reviewed source tests all three jointly on a high-cardinality classifier head with post-refit calibration metrics, so this section reports what each group establishes and the analysis section assembles them into a measurement design.

On teacher temperature, the literature treats it as consequential but usually fixes it by grid search. Li and coauthors argue that a constant temperature means a fixed level of task difficulty, which is suboptimal for a student whose ability grows during training, and propose Curriculum Temperature (CTKD), a dynamic and learnable temperature following an easy-to-hard schedule that plugs into existing frameworks [11]. Through the DKD lens, the teacher temperature is a dial on the NCKD weight: raising T_t moves mass from the target class to the non-target tail, strengthening the NCKD signal until the tail becomes near-uniform noise. The label smoothing literature adds a caution. Muller and coauthors show that label smoothing improves model calibration, but that a teacher trained with label smoothing makes distillation much less effective, because smoothing collapses same-class representations into tight clusters and destroys the information in the logits about resemblances between different classes [7]. An over-softened teacher is the same failure from a different direction: too much softness erases the dark knowledge soft targets exist to carry. These sources imply an interior optimum for teacher softness, though none locates it for decision heads.

On teacher calibration, three sources converge on a finding the older literature missed: the teacher’s calibration error predicts distillation success, and calibrating the teacher before distilling improves the student. Kim and coauthors report a strong correlation between the teacher’s calibration error and the student’s accuracy, and show that applying a calibration method that reduces the teacher’s error improves distillation across classification and detection tasks [8]. Yang and Song revisit the older puzzle (larger teachers make worse teachers) from a calibration perspective: the larger teacher is over-confident so the student cannot imitate it, and after simple calibration of the teacher, teacher size correlates positively with student performance [9]. Yang and coauthors identify a related failure they call mis-instruction, where the student is misled whenever the teacher’s prediction disagrees with the ground-truth label, and propose Logit Calibration (LoCa), which corrects the teacher logits against the labels while preserving remaining dark knowledge such as class discriminability [10]. These cohere with the capacity-mismatch finding of Cho and Hariharan, who report that larger models often do not make better teachers and that stopping the teacher’s training early mitigates the effect [6]: an early-stopped teacher is a less over-confident teacher, consistent with the calibration view. Teacher temperature scaling on held-out labeled data is cheap, leaves the teacher’s argmax unchanged, and has three sources supporting it as a pre-distillation step.

On KL direction, the evidence comes from generative language models rather than classifier heads and must be transferred with care. Gu and coauthors replace forward KL with reverse KL for white-box large language model (LLM) distillation, arguing it prevents the student from overestimating low-probability regions of the teacher distribution, and report that their MiniLLM students (spanning 120M to 13B parameters) generate more precise responses with better calibration among other gains [3]. Wen and coauthors generalize sequence-level distillation to f-divergence minimization, show that existing sequence-level knowledge distillation (SeqKD) and ENGINE approaches are approximations of their framework, and report across four datasets that symmetric distilling losses better force the student to learn the teacher distribution [12]. Neither result concerns a fixed-cardinality softmax head. For a choice head the analogous tradeoff is that forward KL preserves the full tail (good for dark knowledge, possibly at the cost of a flattened top probability) while reverse KL sharpens onto the teacher’s peaks (good for decisiveness, at the cost of discarding the tail). Which direction gives better Brier score and ECE after a refit, at matched accuracy, is UNVERIFIED for decision heads in every source reviewed here.

Two further sources bound the student side. Desai and Durrett report that BERT and RoBERTa are calibrated in-domain out of the box, with out-of-domain calibration error as much as 3.5x lower than baselines, and that temperature scaling further reduces in-domain error while label smoothing helps out-of-domain [13]. Thulasidasan and coauthors report that networks trained with mixup are significantly better calibrated than regularly trained networks across architectures and datasets including ImageNet [16]. And Beyer and coauthors, studying the implicit design choices that make distillation work in vision, demonstrate the recipe’s sensitivity by reaching 82.8% top-1 ImageNet accuracy with a ResNet-50 student, a reminder that the settings this node asks about are first-order [17].

Finding	Source	Setting	Implication for choice heads
Temperature reveals dark knowledge in the teacher softmax	Hinton et al. [1]	MNIST, speech acoustic model	Defines `T_t`; the decision-head value is unablated
Fixed temperature is suboptimal; a dynamic curriculum helps	Li et al., CTKD [11]	Image classification	Test a temperature schedule, not just grid values
NCKD is the prominent reason logit distillation works	Zhao et al., DKD [5]	CIFAR-100, ImageNet, MS-COCO	`T_t` is an NCKD dial; 77-way heads are NCKD-dominated
Smoothing helps calibration but smoothed teachers distill poorly	Muller et al. [7]	Image, translation, speech	Interior optimum for softness
KD is learned label smoothing; weak teachers still help	Yuan et al. [4]	Image classification	Softness carries signal, bounded by [7]
Teacher calibration error predicts student accuracy	Kim et al. [8]	Classification, detection	Calibrate the teacher before distilling
Calibration reverses the big-teacher curse	Yang and Song [9]	Image classification	Teacher temperature scaling first, then distill
Mis-instruction: teacher errors mislead; LoCa corrects them	Yang et al. [10]	Image classification, text generation	Add a LoCa-style arm when labels exist
Big teachers often worse; early stopping mitigates	Cho and Hariharan [6]	Various architectures	Consistent with the calibration view of [8][9]
Reverse KL suits generative LMs; MiniLLM reports better calibration	Gu et al. [3]	Instruction following, 120M to 13B	Direction matters; transfer to fixed heads UNVERIFIED
Symmetric f-divergence losses match the teacher better	Wen et al. [12]	Four generation datasets	Test symmetric losses alongside forward and reverse
Pretrained transformers calibrated in-domain; scaling helps	Desai and Durrett [13]	Natural language inference, paraphrase, commonsense	Student refit stays necessary; in-domain outlook good
Mixup training improves calibration	Thulasidasan et al. [16]	Image incl. ImageNet	Candidate student-side complement, not a refit substitute
Implicit distillation choices are first-order; 82.8% ResNet-50	Beyer et al. [17]	Vision, incl. ImageNet	This node’s settings are not fine-tuning details
Brier mixes calibration and resolution	scikit-learn docs [15]	General classifiers	Matched-accuracy qualifier is required

Analysis

Five conclusions follow, in decreasing order of confidence. The first three are direct readings of the evidence. The last two are measurement designs the evidence motivates but does not itself run.

First, calibrate the teacher before distilling. Three independent sources support this from different angles: correlation of teacher calibration error with student accuracy plus gains from reducing it [8], reversal of the big-teacher curse after calibration [9], and correction of teacher errors against labels while keeping dark knowledge [10]. Teacher temperature scaling on held-out labeled data costs one parameter fit and leaves the teacher’s argmax unchanged, so it changes only the softness and honesty of the soft targets, not which class they point at. A LoCa-style correction arm, which additionally fixes teacher predictions that contradict the labels, is worth including wherever the distillation set is labeled.

Second, the teacher temperature has an interior optimum set by two opposing forces, and the DKD decomposition says where to look. Raising T_t strengthens the NCKD signal that Zhao and coauthors identify as the main reason logit distillation works [5], but pushing softness too far destroys inter-class resemblance information, the failure Muller and coauthors diagnose in label-smoothed teachers [7]. For high-cardinality heads the optimum should sit higher than for few-class heads, because the non-target tail holds nearly all the dark knowledge, but the exact value is UNVERIFIED and CTKD suggests a fixed value may lose to a schedule [11]. The honest grid is therefore two-dimensional from the start: static temperatures crossed with at least one dynamic schedule.

Third, the KL direction needs a three-arm ablation, not a default. Forward KL is the correct default in the narrow sense: it is the standard objective and what the PyTorch implementation computes [14], it is mode-covering so it preserves the tail dark knowledge that high-cardinality heads depend on, and nothing reviewed here disqualifies it for classifier heads. But the generative-model evidence shows direction is first-order: reverse KL changed MiniLLM’s behavior including its calibration [3], and symmetric losses matched the teacher distribution better than one-sided ones in the f-DISTILL experiments [12]. The decision-head analogue is a clean experiment: forward, reverse, and a symmetric mean of the two, each followed by the identical held-out refit, compared on post-refit Brier and ECE at matched accuracy. If the refit erases the direction differences, that is itself the finding: direction would matter only pre-refit.

Fourth, the temperature and direction factors must be crossed, and the run count is affordable. A 4-by-3 grid (four teacher temperatures, three directions) is 12 student trainings on a small encoder, with teacher logits cached once and reused across all cells. The teacher-calibration factor (raw versus scaled teacher) doubles this to 24 runs if fully crossed, or it can run as a second-stage comparison on the best temperature-direction cell. Every cell must report accuracy, Brier score, and ECE after the identical refit protocol, because a cell can win on Brier through accuracy alone while losing on calibration [15]. Matched-accuracy comparison can use calibration metrics within accuracy bands or interpolate settings to a shared operating point; the matching method matters less than refusing to compare cells whose accuracies differ by more than the band width.

Fifth, teachers reachable only through an application programming interface (API) collapse part of this design space and need their own protocol. If the teacher exposes only top-1 decisions or unnormalized scores, T_t is not controllable and reverse KL has no meaningful teacher distribution to seek modes of; the available moves are sampling-based soft-target estimation and response-based transfer of hard decisions. Whether sampled soft targets carry usable dark knowledge for a 77-way head is UNVERIFIED, and any such protocol must budget teacher-query cost per decision explicitly. This constraint binds hardest for closed teachers and is why the open questions below separate the white-box ablation from the API-only case.

Implications for small decision models

For Jev, the implication is that the interesting half of this node is unreachable from outside the API. Jev is closed, so a distiller cannot set its temperature, read its full logits, or choose a KL direction against its true distribution; only response-based transfer over sampled decisions is available. Laya’s BENCHMARKS.md headline table, which warns that its Jev figures are third-party published and never measured in that repository, lists Jev’s ECE after temperature fitting at 0.246 against Laya’s 0.081, with typed-decisions accuracy 0.727 against 0.766 and DAIR Emotion 0.480 against 0.600 [18]. The Jev survey gives the fuller behavioral picture. The point for this node is narrower: a teacher with that calibration profile is exactly where Kim et al. [8] and Yang and Song [9] predict weak distillation, and the prescribed remedy (calibrate the teacher first) is unavailable for a closed API. Distilling Jev responses inherits Jev’s miscalibration unless the student pipeline includes its own refit on labeled held-out data, and the refit needs ground truth rather than more teacher outputs.

For Laya, every knob in this node is available but the teacher is weakest exactly where the node asks about. Laya is open, so its logits, temperatures, and full distributions are accessible, and its per-bucket temperature practice (buckets 2, 3 to 5, 6 to 10, and 11 plus, clamped to [0.5, 5.0], with choice:11+ served at 0.5 rather than its fitted 0.1006) is the closest decision-model precedent for the student-side refit [18]. The Laya teardown documents why Laya struggles at high cardinality: options share a fixed token budget with instructions. A study using Laya as teacher must report high-cardinality accuracy separately rather than averaging a preserved failure into an acceptable mean, and should ask whether the teacher’s 77-way outputs are worth distilling before the option-encoding bottleneck is fixed. Laya’s more natural role in this node’s experiments is the student architecture (open, small, per-bucket refit already implemented) rather than the high-cardinality teacher.

For the proposed Maestro Flash, which has no weights and no results, the implication is a concrete recipe for its distillation stage. The proposal stages training as supervised fine-tuning, then soft-distribution distillation from an open LLM teacher, then proper-scoring-rule reinforcement learning (RL), then held-out temperature fitting. The evidence here suggests five design choices. One, distill from calibrated teacher logits: fit the teacher’s temperature on held-out labeled data first [8][9], since the white-box teacher makes this free. Two, cache teacher logits once and run the temperature-by-direction grid (static T_t values plus one CTKD-style schedule [11], crossed with forward, reverse, and symmetric losses [3][12]). Three, include a LoCa-style arm correcting teacher predictions against ground-truth labels where the distillation set is labeled [10]. Four, refit the student per option-count bucket with clamping, following Laya’s precedent [18], and report Brier, ECE, and accuracy for every cell. Five, measure calibration after every subsequent stage, because the scoring-rule RL stage can move calibration independently, and the proposal’s go/no-go criteria depend on knowing which stage moved it.

Limitations

This survey read abstracts, repository files, and official documentation, not full paper texts or codebases, and it ran no experiments. Every quantitative claim is attributed inline to the source that reported it, and those figures are self-reported by their authors under their own protocols, not independently reproduced here. Cross-paper comparisons are not like-for-like: teachers, tasks, cardinalities, and metrics differ across sources.

Three specific gaps limit what this survey can conclude. First, no reviewed source ablates teacher temperature against KL direction on a fixed-cardinality classifier head with post-refit Brier and ECE at matched accuracy; the KL-direction evidence comes from generative models [3][12] and the temperature evidence from few-class image tasks [11]. Second, the high-cardinality setting is thinly evidenced: DKD’s ImageNet results [5] are the closest large-cardinality logit-distillation data point, and nothing reviewed tests the 77-way decision-head configuration. Third, the API-only teacher protocol (sampled soft targets, query budgets, unnormalized scores) appears in no reviewed source as a tested method, so the fifth analysis conclusion is a design sketch rather than a summary of evidence.

Open questions

What teacher temperature and KL direction minimize post-refit Brier score and ECE on a 77-way choice head at matched accuracy when the teacher is a calibrated LLM and the student refit is per-bucket?
Does calibrating the teacher with temperature scaling or LoCa-style correction before distillation improve the student’s post-refit ECE, or only its accuracy?
Does symmetric or reverse KL preserve non-target dark knowledge on high-cardinality heads better than forward KL after the calibration refit, or does the refit erase the difference?

References

← All papers