Do Calibrated Teachers Produce Calibrated Students

This paper is a child of Teacher Temperature and KL Direction for Calibrated Distillation, which recommended calibrating the teacher before distilling and left the present question open. It answers one clause of that recommendation: whether the gain lands on student accuracy, student calibration, or both. All arXiv identifiers (IDs) below were verified against their arXiv abstract pages, and every number is attributed inline to a source fetched for this paper. Velofy has trained no model for this paper and ships none with it.

The question

Does calibrating the teacher with temperature scaling or a Logit Calibration (LoCa) style correction before distillation improve the student’s post-refit expected calibration error (ECE), or only its accuracy?

The question matters because the parent paper’s recommendation rests on three sources that report student accuracy gains, while the decision-model pipeline this graph studies always refits the student afterward. Kim and coauthors report a strong correlation between the teacher’s calibration error and the student’s accuracy, plus gains from reducing teacher error [3]. Yang and Song report that calibration of an over-confident teacher restores a positive correlation between teacher size and student performance [4]. Yang and coauthors report that LoCa correction of teacher logits improves baseline performance [5]. None of these three abstracts reports the student’s ECE after a held-out student-side refit, so the calibration half of the claim is UNVERIFIED in the sources that motivate it.

The terms need fixed meanings before the evidence can be weighed. Teacher temperature scaling here means fitting one scalar on held-out labeled data after teacher training, the procedure Guo and coauthors describe as a single-parameter variant of Platt scaling that is surprisingly effective on most datasets [2]. It changes only the sharpness of the teacher softmax and leaves the teacher argmax unchanged. LoCa-style correction means adjusting teacher logits against ground-truth labels so that predictions contradicting the labels are corrected while remaining dark knowledge is preserved, the mis-instruction fix Yang and coauthors propose [5]. Unlike temperature scaling, it can change which class the soft target points at. Post-refit ECE means expected calibration error measured after the student’s own held-out temperature fit, the per-bucket refit this graph treats as standard student-side practice. The matched-accuracy qualifier carries over from the parent paper: the scikit-learn calibration documentation warns, following the Brier score decomposition of Murphy, that a lower Brier loss does not necessarily mean a better calibrated model, since the score mixes calibration with discriminative power [12]. Any claim about student ECE across teacher treatments is confounded unless accuracy is held fixed or reported jointly.

Reduced to first principles, the question is about what one scalar can undo. Both temperature fits are the same operation: divide every logit by one number. That changes how sharp the probabilities are, but not which class ranks first or how the other classes are ordered within an example. The student refit is the last step and is fit on held-out data, so any difference between teacher treatments that is only a difference in sharpness should be refit away. A teacher treatment can show up in post-refit ECE only if it changes something a single student scalar cannot reach: the rank structure the student learns, or which class the soft target favors. Teacher temperature scaling keeps both the class order and the favored class of each soft target, so any effect it has must come through what the student learns from the reshaped targets. LoCa can change the favored class directly. The analysis below follows from this asymmetry.

Background

Knowledge distillation trains a student against a teacher’s softened output distribution alongside ordinary supervised loss. Hinton and coauthors introduced the temperature parameter that softens the teacher softmax so that the relative probabilities of non-target classes survive in the training signal [1]. The parent paper on this site decomposes that signal further and should be read first; this paper does not repeat its method tables. The one background point worth restating is that a high-cardinality head concentrates nearly all soft-target mass on non-target classes, so treatments that reshape the teacher distribution act mostly on the tail.

Calibration is the axis this node isolates. Guo and coauthors found that modern networks are poorly calibrated and reported that single-parameter temperature scaling on held-out data is surprisingly effective [2]. ECE measures the gap between predicted probabilities and empirical accuracies across confidence bins, but the metric is less settled than its popularity suggests. Nixon and coauthors report, across MNIST, Fashion MNIST, CIFAR-10/100, and ImageNet, that conclusions on the rank ordering of recalibration methods are drastically impacted by the choice of calibration measure, and they recommend conditioning on the class, using the L2 norm rather than L1, and using adaptive binning schemes [8]. Kull and coauthors generalize this concern to the multiclass case: they propose Dirichlet calibration as a natively multiclass method and evaluate on confidence-ECE, classwise-ECE, log-loss, and Brier score across datasets and classifiers [9]. For decision heads these warnings are load bearing. A treatment that improves confidence-ECE (which looks only at the top probability) may leave classwise-ECE (which looks at every class) unchanged, and on a head with dozens of options the tail is where distillation lives.

Two student-side facts bound the question from the other side. Desai and Durrett report that BERT and RoBERTa are calibrated in-domain out of the box, with out-of-domain calibration error as much as 3.5x lower than baselines, and that temperature scaling further reduces in-domain error while label smoothing helps out-of-domain [10]. Thulasidasan and coauthors report that networks trained with mixup are significantly better calibrated than regularly trained networks across architectures and datasets including ImageNet, and they attribute much of the benefit to the label smoothing component of mixup [11]. Both sources show that student-side treatments move student calibration without touching the teacher, which sets up the central ambiguity: if the student refit already fixes ECE, teacher calibration may add accuracy only.

One more background result constrains which teacher treatments are even candidates. Muller and coauthors show that label smoothing improves model calibration but that a teacher trained with label smoothing makes distillation much less effective, because smoothing collapses same-class representations into tight clusters and destroys the inter-class resemblance information in the logits [7]. Teacher calibration achieved by changing training (smoothing, heavy regularization) can therefore hurt teaching even as it helps the teacher’s own ECE. Teacher temperature scaling avoids this failure by construction: it is fit after training, touches no representations, and only rescales logits. Whether that distinction is the full story is part of what the analysis section takes up.

What the evidence says

The evidence falls into three groups: teacher calibration improving student accuracy, student calibration achieved without teacher calibration, and measurement results that qualify both. No reviewed source runs the exact experiment this node asks for (teacher calibration treatment crossed with student refit on versus off, reporting post-refit student ECE at matched accuracy), so each group bears on the question indirectly.

On teacher calibration and student accuracy, the three parent-paper sources agree. Kim and coauthors report a strong correlation between teacher calibration error and student accuracy and demonstrate gains from a calibration method that reduces teacher error, across tasks from classification to detection, including integration with existing state-of-the-art methods [3]. Yang and Song revisit the puzzle that larger teachers make worse teachers and attribute it to over-confidence: the student cannot effectively imitate an over-confident teacher, and after simple calibration of the teacher, teacher size correlates positively with student performance [4]. Yang and coauthors identify mis-instruction, where the student is misled whenever teacher-logit predictions disagree with the labels, and propose LoCa, which calibrates teacher logits against ground truth while preserving dark knowledge such as class discriminability, with reported gains on image classification and text generation tasks [5]. The abstracts report performance and accuracy outcomes, not student-side post-refit ECE. That absence is the gap this paper exists to name: the recommendation to calibrate the teacher is well supported as an accuracy intervention and UNVERIFIED as a post-refit calibration intervention.

On student calibration without teacher calibration, one source is directly on point and two are supporting. Mishra and coauthors study how to obtain a calibrated student from an uncalibrated teacher by fusing augmentation techniques (cutout, mixup, CutMix) with distillation, extend the approach to relational and contrastive distillation variants, and report calibrated student models without compromising distilled accuracy across CIFAR-10, CIFAR-100, CINIC-10, and TinyImageNet, with robust performance on corrupted CIFAR-100C data [6]. This is the closest published answer to the present question, and its direction is cautionary: student calibration was achieved student-side, with the teacher left uncalibrated. Guo and coauthors provide the refit half of the same point: post-hoc temperature scaling on held-out data is surprisingly effective, which means a student-side scalar fit already removes the global over-confidence component of ECE regardless of what the teacher looked like [2]. Desai and Durrett add that pre-trained transformers start calibrated in-domain and that temperature scaling reduces error further [10]. Together these sources suggest the refit is doing real calibration work that teacher treatment may not need to duplicate.

On measurement, three sources say the outcome variable itself needs care. Nixon and coauthors show that the choice of calibration measure changes the rank ordering of recalibration methods, which means a teacher-calibration study reporting only one ECE variant could reach a different conclusion under another [8]. Kull and coauthors show that confidence-ECE and classwise-ECE are distinct measures and that a natively multiclass calibrator can improve both alongside log-loss and Brier score [9]. The scikit-learn documentation warns that Brier score mixes calibration, resolution, and uncertainty, so a lower Brier loss can reflect discriminative power rather than calibration [12]. Any test of this node’s question must therefore fix the ECE variant in advance (at minimum confidence-ECE and classwise-ECE), report accuracy jointly, and compare treatments within accuracy bands.

Finding	Source	Reported outcome	Gap for this node
Teacher calibration error predicts student accuracy; reducing it helps	Kim et al. [3]	Student accuracy, classification to detection	Student ECE, pre- or post-refit, UNVERIFIED
Calibration reverses the big-teacher curse	Yang and Song [4]	Student performance vs teacher size	Student ECE UNVERIFIED
LoCa corrects teacher mis-instruction against labels	Yang et al. [5]	Baseline performance, image and text	Student ECE UNVERIFIED; correction vs rescaling unseparated
Calibrated student from uncalibrated teacher via augmentation fusion	Mishra et al. [6]	Student calibration without accuracy loss, CIFAR-10/100, CINIC-10, TinyImageNet, CIFAR-100C	Teacher-calibration arm missing; refit interaction untested
Temperature scaling surprisingly effective post-hoc	Guo et al. [2]	Calibrated predictions, image and document data	Applied to student, it may erase teacher sharpness differences
Pretrained transformers calibrated in-domain; scaling helps further	Desai and Durrett [10]	In-domain calibration, out-of-domain error as much as 3.5x lower	Decision-head transfer UNVERIFIED
Mixup training improves calibration via label smoothing	Thulasidasan et al. [11]	Calibration across architectures incl. ImageNet	Student-side only; says nothing about the teacher
Smoothed teachers distill poorly despite good calibration	Muller et al. [7]	Distillation efficacy vs teacher smoothing	Separates calibration-by-training (harmful) from calibration-by-rescaling (safe)
ECE variant choice changes method rankings	Nixon et al. [8]	Rank order across MNIST, Fashion MNIST, CIFAR-10/100, ImageNet	Mandates pre-registered ECE variants for this node’s test
Confidence-ECE and classwise-ECE are distinct targets	Kull et al. [9]	Multiple measures across datasets and classifiers	High-cardinality heads need classwise reporting
Brier mixes calibration and resolution	scikit-learn docs [12]	General classifiers	Matched-accuracy comparison required

Analysis

Four conclusions follow, ordered by confidence. The first two are mechanism decompositions that convert the question into falsifiable predictions. The third is the experiment that would settle it. The fourth is the confounder that could invalidate a naive reading of existing results.

First, pure teacher temperature scaling should be expected to improve student accuracy more than student post-refit ECE, because the student refit undoes exactly what teacher scaling does. Teacher scaling multiplies teacher logits by one scalar, preserving argmax and changing only sharpness. The student refit multiplies student logits by one fitted scalar, also preserving argmax. Any global sharpness the student inherits from the teacher is therefore re-optimized on held-out data after distillation, and two students distilled from differently scaled teachers should converge to nearby post-refit sharpness if their rank orderings agree. The surviving difference, if any, must live in rank structure: relative tail orderings or inter-class gaps that a single student scalar cannot replicate. Whether teacher scaling changes those in a way that matters for post-refit ECE is UNVERIFIED, but the mechanism says the default prediction is accuracy gains with post-refit ECE unchanged. This reading is consistent with all three teacher-calibration sources reporting accuracy-style outcomes [3][4][5] and with Guo and coauthors on the power of the student-side scalar itself [2].

Second, LoCa-style correction has a stronger mechanism for surviving the refit than temperature scaling, because it changes soft-target argmax on teacher-error cases. Correcting a teacher prediction that contradicts the label moves probability mass across classes, not just sharpness within a fixed ranking, and no student-side temperature can replicate a mass transfer it never saw. If mis-instruction measurably distorts the student’s learned decision boundaries, the LoCa student should show better post-refit ECE (fewer confidently wrong predictions on the corrected cases) in addition to better accuracy. But this is a mechanism argument, not an established result: Yang and coauthors report performance gains, not post-refit ECE [5], and the correction could in principle help accuracy while leaving calibration to the refit. The honest statement is that LoCa is the teacher treatment most likely to move post-refit ECE, and the magnitude is UNVERIFIED.

Third, the experiment that settles the question is a small factorial grid, and it is affordable because teacher logits are cached once. Three teacher arms (raw logits, temperature-scaled logits, LoCa-corrected logits) crossed with two student conditions (ECE measured before and after the identical held-out refit) gives six cells per teacher scale, each reporting accuracy, confidence-ECE, classwise-ECE, and Brier score. The teacher-calibration split and the student-refit split must be disjoint held-out partitions, or the refit will overfit the same data that scaled the teacher. Comparisons must be made within accuracy bands per the scikit-learn warning [12], and ECE variants must be pre-registered per Nixon and coauthors [8], with classwise-ECE mandatory at high cardinality per Kull and coauthors [9]. A seventh cell, distilling from the raw teacher with Mishra-style augmentation fusion [6], would separate the teacher-calibration hypothesis from the student-side alternative in one run. The predicted signature of the mechanism above is: teacher scaling wins pre-refit ECE and accuracy but ties post-refit ECE; LoCa wins post-refit ECE as well; augmentation fusion matches teacher scaling on post-refit ECE without any teacher treatment. Any other signature is informative, and a full tie post-refit is itself the finding (teacher calibration would be an accuracy intervention only).

Fourth, the confounder in existing results is teacher accuracy, which covaries with teacher calibration in every cited source. LoCa correction against labels raises the teacher’s effective correctness on the distillation set; temperature scaling fit on held-out data does not change argmax but is typically applied to teachers whose accuracy differs across comparison arms. If better-calibrated teachers in these studies are also more accurate teachers, the student accuracy gains could reflect cleaner supervision rather than better-calibrated uncertainty. The matched-accuracy discipline cuts both ways here: just as Brier comparisons need accuracy held fixed [12], teacher-calibration comparisons need teacher accuracy reported per arm, or the student gains cannot be attributed to calibration as distinct from correctness. This is a further reason the existing accuracy gains do not imply post-refit ECE gains.

Implications for small decision models

For Jev, the implication is that teacher calibration is unavailable and the student refit must carry all of the calibration load. Jev is a closed application programming interface (API), so a distiller cannot fit its temperature, read its full logits, or apply LoCa-style correction against labels; only response-based transfer over sampled decisions is available. The calibration profile of such a teacher is unpromising: Laya’s BENCHMARKS.md headline table, which warns that its Jev figures are third-party published and never measured in that repository, lists Jev’s ECE after temperature fitting at 0.246 against Laya’s 0.081, with typed-decisions accuracy 0.727 against 0.766 and DAIR Emotion 0.480 against 0.600 [13]. The Jev survey gives the fuller behavioral picture. For this node the narrow point is that distilling Jev responses inherits Jev’s miscalibration in the soft targets, teacher-side remedies are all blocked by the API boundary, and the only remaining calibration lever is the student’s own refit on labeled held-out data with ground truth rather than more teacher outputs. Whether that refit fully erases the teacher’s miscalibration is exactly the experiment the analysis section specifies, run with a closed teacher.

For Laya, the implication is that it is better instrumented as the student than as the teacher in this node’s experiment. Laya is open: its logits, temperatures, and full distributions are accessible, and its checkpoints are published under the convaiinnovations organization on Hugging Face, which a model search for laya returns alongside community conversions [14]. Its per-bucket temperature practice, with the affected high-cardinality bucket now served at 0.5 rather than its fitted 0.1006 after the clamp to [0.5, 5], is the closest decision-model precedent for the student-side refit [13]. But the Laya teardown documents why Laya struggles as a high-cardinality teacher: options share a fixed token budget with instructions, and the teardown reports Banking77 at 0.425 for both English and multilingual checkpoints. A study using Laya as teacher must report high-cardinality accuracy separately and ask whether its outputs at that cardinality are worth distilling before the option-encoding bottleneck is fixed. Laya’s Khmer failure, 0.000 accuracy at 0.952 confidence per its BENCHMARKS.md [13], is also a useful stress case for the LoCa arm: a teacher that is confidently wrong is where label correction should help most, and where pure rescaling helps least.

For the proposed Maestro Flash, which has no weights and no results, the implication is an ordering constraint on its distillation stage rather than a new component. The proposal stages training as supervised fine-tuning, then soft-distribution distillation from an open large language model (LLM) teacher, then proper-scoring-rule reinforcement learning (RL), then held-out temperature fitting. The evidence here suggests four design choices. One, calibrate the teacher first anyway: it is one scalar fit on held-out data, it leaves argmax unchanged, and three sources support accuracy gains [3][4][5], so there is no reason to distill from a knowingly over-confident teacher. Two, include a LoCa-style arm wherever the distillation set is labeled, since correction is the treatment with the strongest mechanism for surviving the student refit. Three, never report teacher-calibration effects on student ECE without the refit ablation: every teacher arm must be measured before and after the identical student refit, or the paper will repeat the literature’s gap of reporting accuracy only. Four, fix the ECE variants and accuracy bands before running, following Nixon and coauthors on measure sensitivity [8] and the scikit-learn documentation on Brier confounding [12], because post-hoc choice among ECE variants can manufacture whichever conclusion the experimenter prefers.

Limitations

This survey read abstracts, repository files, model search results, and official documentation, not full paper texts or codebases, and it ran no experiments. Every quantitative claim is attributed inline to the source that reported it, and those figures are self-reported by their authors under their own protocols, not independently reproduced here. Cross-paper comparisons are not like-for-like: teachers, tasks, cardinalities, and metrics differ across sources.

Four specific gaps limit what this survey can conclude. First, no reviewed source crosses teacher calibration treatment with student refit on versus off while reporting post-refit student ECE at matched accuracy, so the central question remains UNVERIFIED and the analysis predictions are mechanisms awaiting a run. Second, the teacher-calibration abstracts report accuracy-style outcomes without the per-arm teacher accuracy and student ECE detail needed to separate calibration effects from correctness effects. Third, the ECE measurement literature [8][9] implies that any future result on this question is conditional on the chosen variants, bins, and norms, which no decision-model study on this site has yet fixed. Fourth, the high-cardinality decision-head setting is thinly evidenced throughout: the closest data points are multiclass image tasks, and nothing reviewed tests the option-count regime the parent paper studies.

Open questions

Does LoCa-style teacher correction improve student post-refit classwise-ECE at matched accuracy where pure teacher temperature scaling does not?
What is the smallest teacher-student experiment that separates teacher-calibration effects from student-refit effects on post-refit ECE at high option counts?
Do ECE variant choices change the ranking of teacher-calibration treatments for decision heads the way Nixon and coauthors report for recalibration methods?

References

← All papers