Symmetric and Reverse KL for Tail Dark Knowledge

This paper is a child of Teacher Temperature and KL Direction for Calibrated Distillation, which left this exact question open as its third open question. It builds on that paper and on its sibling about teacher calibration before distillation without repeating their method tables, benchmark summaries, or measurement warnings. All arXiv identifiers (IDs) below were verified against their arXiv abstract pages, every number is attributed inline to a source fetched for this paper, and Velofy has trained no model for this paper and ships none with it.

The question

Does symmetric or reverse Kullback-Leibler (KL) divergence preserve non-target dark knowledge on high-cardinality heads better than forward KL after the calibration refit, or does the refit erase the difference?

The question names three objects and one operation: the KL direction of the distillation loss (forward, reverse, or a symmetric combination), the non-target dark knowledge (the relative probabilities of wrong answers, which Zhao and coauthors isolate as non-target class knowledge distillation (NCKD) and identify as the prominent reason logit distillation works [2]), and high-cardinality choice heads (on the order of 77 options, where nearly all soft-target mass sits on non-target classes). The operation is the calibration refit: a held-out temperature fit applied to the student after distillation, in the per-bucket form this graph treats as standard. The comparison inherits the matched-accuracy qualifier from the parent paper, because the scikit-learn calibration documentation warns, following the Brier score decomposition of Murphy, that a lower Brier loss does not necessarily mean a better calibrated model, since the score mixes calibration with discriminative power [16].

Background

Knowledge distillation (KD) trains a student to match the softened output distribution of a teacher alongside ordinary supervised loss. Hinton and coauthors introduced the temperature parameter that softens the teacher softmax so that information about class similarities survives in the training signal, and their paper already gestures at fine-grained structure through specialist models for classes the full models confuse [1]. Zhao and coauthors split the classical KD loss into target-class knowledge distillation (TCKD), which transfers sample difficulty, and NCKD, which carries the inter-class resemblance signal; they further show that the classical coupled formulation suppresses NCKD, and propose Decoupled Knowledge Distillation (DKD) to weight the two parts independently [2]. For a 77-way head this split is the whole game: with 76 non-target classes, NCKD holds nearly all of the dark knowledge, and any loss choice that discards the tail discards nearly all of the point of distillation.

The three directions differ in which distribution weights the penalty. The PyTorch KLDivLoss documentation defines the pointwise loss as target * (target.log() - input), with the teacher distribution as target, and warns that only reduction="batchmean" returns the mathematically correct KL value [13]. That is forward KL: the teacher weights the loss, so the student spreads mass to cover the teacher’s support (mode-covering). Reverse KL swaps the roles, so the student concentrates on the teacher’s peaks and may ignore low-probability regions (mode-seeking). The symmetric family penalizes both failures at once: the plain mean of the two directions, the Jensen-Shannon (JS) and total variation distance (TVD) distillations in f-DISTILL [4], the dual-sided penalty of DistillLens [11], and the skew KL losses of DistiLLM, where an alpha parameter controls the teacher-student mixing ratio inside the divergence [6]. The parent paper covers teacher temperature and the interior optimum for softness; the sibling paper covers teacher calibration treatments, disjoint data splits, and expected calibration error (ECE) variant sensitivity. None of that is repeated here.

The refit is a single-parameter rescaling fit on held-out data. Guo and coauthors report that temperature scaling, a single-parameter variant of Platt scaling, is surprisingly effective at calibrating predictions on most datasets [12]. Rescaling is monotonic in the logits: it changes sharpness but preserves class rankings and leaves the argmax unchanged. That fact drives the analysis below. Direction differences in global sharpness should be erased by the refit, since each student gets its own fitted scalar afterward. Direction differences in learned rank structure (tail orderings, inter-class gaps) should survive, since no scalar restores orderings the student never learned. Laya’s per-bucket refit is the closest decision-model precedent: its BENCHMARKS.md reports mean ECE falling from 0.466 as shipped to 0.081 after refitting one temperature per question type and option count on held-out data [17]. The Laya teardown documents the architecture behind those numbers.

What the evidence says

The evidence falls into four groups: reverse KL discarding the tail by design, symmetric losses matching the teacher better, the tail carrying the signal on classification heads, and the refit fixing sharpness rather than structure. No reviewed source combines KL direction with a student refit on a fixed-cardinality head, so the node question is UNVERIFIED in every source reviewed here.

On reverse KL and the tail, two sources are explicit that tail-dropping is the mechanism, not a side effect. Gu and coauthors replace forward KL with reverse KL for white-box large language model (LLM) distillation, arguing it prevents the student from overestimating low-probability regions of the teacher distribution, and report MiniLLM students spanning 120M to 13B parameters with better calibration among other gains [3]. The project README describes the method as minimizing reverse KLD by on-policy distillation [14], and a model listing for the MiniLLM organization returned 50 models at the time of this writing, including the students and their teachers (gpt2-1.5B, OPT-13B, Llama-13B) [15]. Wen and coauthors state the same geometry from the framework side: reverse KL produces mode collapse onto high-probability regions, while forward KL produces mode averaging [4]. For a generative model this favors reverse KL, because the tail over a vocabulary is mostly noise. For a 77-way head the sign flips: the tail is the NCKD signal that Zhao and coauthors identify as the prominent reason logit distillation works [2]. Transferring the MiniLLM lesson to heads requires an argument no reviewed source makes.

On symmetric losses, four sources converge on matching the teacher distribution more completely, all in sequence or generation settings. Wen and coauthors formulate sequence-level distillation as f-divergence minimization with four variants, show SeqKD and ENGINE are approximations of KL and reverse KL distillation, and derive symmetric Jensen-Shannon and total variation distillations that alleviate both mode averaging and mode collapsing; experiments across four datasets show the symmetric losses better force the student to learn the teacher distribution [4]. Dhakal and coauthors introduce DistillLens, which aligns student and teacher hidden states projected into vocabulary space with a symmetric divergence objective; their analysis says the dual-sided penalty prevents both overconfidence and underconfidence while preserving the high-entropy conduits used for final deduction, with experiments on GPT-2 and Llama [11]. Ko and coauthors introduce DistiLLM with a skew KL loss whose alpha parameter controls the teacher-student mixing ratio, plus a mirrored skew reverse KL and a generalized Jensen-Shannon divergence with interpolation parameter beta; they report up to 4.3x speedup compared to recent KD methods, with code published alongside [6][18]. Agarwal and coauthors introduce Generalized Knowledge Distillation (GKD), training on self-generated sequences with teacher feedback, and note the flexibility to employ alternative loss functions when the student cannot mimic the teacher distribution [5]. Every one of these results concerns generative models. Whether symmetric matching preserves head-level tail rankings better than forward KL after a refit is UNVERIFIED.

On the tail as signal, two classifier-head precedents say the tail is worth preserving, and one complicates the direction story. Zhao and coauthors report DKD reaching results comparable to or better than complex feature-based methods on CIFAR-100, ImageNet, and MS-COCO [2]. ImageNet is a 1000-way head, which makes DKD the closest large-cardinality logit-distillation data point to the decision-head regime. Touvron and coauthors provide a second precedent: their reference 86M-parameter vision transformer reaches 83.1% top-1 ImageNet accuracy with no external data, trained on a single computer in less than 3 days, and their teacher-student strategy with a distillation token reaches up to 85.2% [10]. The complication comes from Malinin and Gales, who show that reverse KL between Dirichlet distributions is the appropriate Prior Networks criterion, enabling classification tasks with arbitrarily many classes, with experiments on CIFAR-10 and CIFAR-100 [7]. Reverse KL is therefore not inherently anti-high-cardinality. But that result concerns Dirichlet distributions over output distributions, not softmax logit matching, so it qualifies rather than settles the head-level question.

On the refit, three sources plus the decision-model precedent bound what recalibration can erase. Guo and coauthors establish the refit half directly: post-hoc temperature scaling on held-out data is surprisingly effective, so a student-side scalar already removes the global overconfidence component of ECE [12]. Laya’s BENCHMARKS.md makes the point concrete: the temperature refit moves mean ECE from 0.466 to 0.081, below the 0.246 it lists for Jev, and the file warns that its Jev figures are third-party published and never measured in that repository [17]. Sun and coauthors show why surviving differences must live in relations rather than magnitudes: a shared teacher-student temperature forces an exact match of logit range and variance, while their Z-score logit standardization lets the student focus on the teacher’s logit relations instead, with evaluations on CIFAR-100 and ImageNet [8]. Stanton and coauthors add two cautions for any null result: a surprisingly large teacher-student discrepancy often remains even when the student has capacity to match the teacher, with optimization difficulties as a key reason, and closer matching paradoxically does not always improve generalization [9]. A post-refit tie could therefore mean the refit erased a real difference, or that the student never matched any teacher distribution closely enough for direction to bind.

Finding	Source	Setting	Implication for the node question
Dark knowledge lives in softened non-target probabilities	Hinton et al. [1]	MNIST, speech	Defines the tail signal
NCKD is the prominent reason logit distillation works	Zhao et al., DKD [2]	CIFAR-100, ImageNet, MS-COCO	77-way heads are NCKD-dominated
Reverse KL drops low-probability regions; better calibration	Gu et al., MiniLLM [3]	Instruction following, 120M to 13B	Head transfer UNVERIFIED
Reverse KL collapses; forward KL averages	Wen et al. [4]	Framework, four generation datasets	Direction geometry confirmed
Symmetric JS and TVD losses match the teacher better	Wen et al. [4]	Four generation datasets	Test symmetric losses too
Dual-sided penalty preserves high-entropy conduits	Dhakal et al., DistillLens [11]	GPT-2 and Llama	Head transfer UNVERIFIED
Skew KL with mixing alpha; up to 4.3x speedup	Ko et al., DistiLLM [6]	Instruction following and more	Skew sweep is a direction dial
GKD allows alternative student-teacher losses	Agarwal et al. [5]	Summarization, translation, reasoning	Direction is a supported axis
Reverse KL between Dirichlets scales to many classes	Malinin and Gales [7]	CIFAR-10, CIFAR-100	Dirichlet result, not softmax
Logit relations matter more than magnitude matching	Sun et al. [8]	CIFAR-100, ImageNet	Refit fixes magnitude only
Students often fail to match teachers; closer can generalize worse	Stanton et al. [9]	Discrepancy study	Nulls need a manipulation check
Distillation token scales to 1000-way heads; up to 85.2%	Touvron et al., DeiT [10]	ImageNet, one computer, under 3 days	High-cardinality precedent
Temperature scaling surprisingly effective post-hoc	Guo et al. [12]	Image and document classification	Sharpness gaps should not survive
Refit moves ECE 0.466 to 0.081, below Jev at 0.246	Laya BENCHMARKS.md [17]	Typed decisions, per-bucket refit	Refit dominates ECE
Forward KL definition; only batchmean is correct	PyTorch docs [13]	General classifiers	Fixes forward arm code
Brier mixes calibration, resolution, uncertainty	scikit-learn docs [16]	General classifiers	Matched accuracy required

Analysis

Four conclusions follow, ordered by confidence. The first two are mechanism predictions the evidence supports but no reviewed source tests on heads. The third is the experiment that would settle the question. The fourth names the confounders that could invalidate a naive reading of the result.

First, the pre-refit ranking for tail preservation should be forward and symmetric ahead of reverse, with symmetric plausibly best. Forward KL averages (covers the tail, possibly flattening the top probability) while reverse KL collapses onto high-probability regions [4], and Gu and coauthors confirm the reverse-KL side empirically in generation, where discarding low-probability regions helps [3]. On a head where the tail is the NCKD signal [2], that discarding removes supervision the student needs. Symmetric losses penalize both failures [4][11], and DistiLLM’s skew family interpolates the mixture continuously [6]. The open part is whether symmetric matching preserves more usable tail rank than forward KL or merely matches teacher noise more tightly (tighter matching can generalize worse [9]). The predicted pre-refit tail-fidelity order is symmetric first, forward second, reverse last.

Second, the refit should erase sharpness differences and preserve ranking differences, which splits the metrics. Confidence-ECE looks only at the top probability, so direction differences there should largely vanish after each student gets its own fitted scalar [12][17]. Classwise-ECE looks at every class, so it reflects tail structure the refit cannot rebuild: relations are what the student must learn because magnitudes are refit afterward [8]. The default prediction is a split signature: all three directions tie on accuracy and confidence-ECE after the identical refit, while forward and symmetric beat reverse on classwise-ECE and on tail probes such as teacher-student rank correlation over non-target classes. A full post-refit tie on every metric is itself the finding (the refit erases the difference). A reverse-KL win on classwise-ECE would contradict the mechanism, since it would mean the mode-seeking student learned tail order it was never penalized for missing.

Third, the experiment that settles the question is a three-arm ablation with a pre-refit manipulation check, affordable because teacher logits are cached once. Three direction arms (forward, reverse, symmetric mean, with a skew midpoint as an optional fourth cell [6]) crossed with two measurement points (before and after the identical per-bucket held-out refit) gives six to eight cells per teacher temperature. Every cell reports accuracy, confidence-ECE, classwise-ECE, Brier score, and at least one tail probe (Spearman correlation between teacher and student non-target rankings, plus top-k overlap for small k). Comparisons are made within accuracy bands [16], ECE variants are pre-registered, and the teacher-calibration split and the student-refit split are disjoint partitions, following the sibling paper. The manipulation check comes from Stanton and coauthors [9]: the arms must separate pre-refit on tail fidelity, or a post-refit tie proves only that the objective never bound. Teacher accuracy is reported per arm, since it covaries with calibration treatments.

Fourth, two deployment constraints collapse this design space. Teachers reachable only through an application programming interface (API) remove the direction choice almost entirely: reverse and symmetric losses need teacher density queries on student-generated outputs (the on-policy requirement in [3] and [5]), which a closed API cannot serve at scale, while forward KL needs only the returned distribution. Whether an API’s returned probabilities carry a usable tail is UNVERIFIED for every closed decision API reviewed on this site. Compute cost cuts the other way: symmetric and skew losses cost roughly twice the divergence evaluations per step, and DistiLLM’s 4.3x speedup claim for its full framework [6] does not imply the skew loss itself is free. If symmetric ties forward post-refit on every metric including classwise-ECE, it is strictly worse engineering (same result, more compute).

Implications for small decision models

For Jev, only the forward arm of this node is reachable, and the tail it learns from may be degraded at the source. Jev is a closed API, so a distiller cannot set its temperature, read its full training logits, or query teacher densities on student samples for reverse or symmetric on-policy losses. The Jev survey gives the fuller behavioral picture. Forward KL against Jev’s returned probability vectors is the only practical direction, and it inherits whatever truncation or rounding the API applies to low probabilities, which is exactly the tail this node cares about. The calibration profile is also unpromising: Laya’s BENCHMARKS.md headline table, which warns that its Jev figures are third-party published and never measured in that repository, lists Jev’s ECE after temperature fitting at 0.246 against Laya’s 0.081, with typed-decisions accuracy 0.727 against 0.766 and DAIR Emotion 0.480 against 0.600 [17]. Distilling Jev responses inherits Jev’s miscalibration, direction ablations are blocked by the API boundary, and the only remaining lever is the student’s own refit on labeled held-out data.

Laya is the natural student and harness for this node’s experiment, not the teacher. Laya is open, so its logits, temperatures, and full distributions are accessible, and its per-bucket refit (with the high-cardinality bucket served at 0.5 rather than its fitted 0.1006 after the clamp to [0.5, 5]) is the exact post-refit measurement stage this question needs [17]. But the Laya teardown documents why Laya struggles as a high-cardinality teacher: options share a fixed token budget with instructions, and both checkpoints score exactly 0.425 on Banking77’s 77 labels, which the benchmarks file attributes to that budget ceiling [17]. A 0.425-accuracy teacher’s tail orderings are mostly orderings among confusions, so distilling from Laya at high cardinality would test whether direction preserves noise, not knowledge. Laya’s correct role is the student architecture distilled from a stronger open LLM teacher, with high-cardinality accuracy reported separately.

For the proposed Maestro Flash, which has no weights and no results, the implication is a concrete direction protocol inside its distillation stage. The proposal stages training as supervised fine-tuning, then soft-distribution distillation from an open LLM teacher, then proper-scoring-rule reinforcement learning (RL), then held-out temperature fitting. The evidence suggests five design choices. One, run the three direction arms (forward, reverse, symmetric) from cached teacher logits, with one skew midpoint if budget allows [6]. Two, evaluate every arm on classwise-ECE and tail probes, not only confidence-ECE, because the analysis predicts the difference survives only there. Three, keep the pre-refit measurement as a manipulation check [9]: arms that never separate pre-refit cannot inform the post-refit comparison. Four, keep the teacher-calibration split and the student-refit split disjoint, following the sibling paper, and pre-register ECE variants and accuracy bands. Five, measure calibration after the scoring-rule RL stage as well, because RL can move the tail the distillation stage just taught.

Limitations

This survey read the arXiv abstract pages for every arXiv ID cited, plus excerpts from the full-text HyperText Markup Language (HTML) versions for the skew KL definitions in DistiLLM [6] and the symmetric variants in f-DISTILL [4], alongside repository files, model listings, and official documentation. It ran no experiments. Every quantitative claim is attributed inline to the source that reported it, and those figures are self-reported by their authors under their own protocols, not independently reproduced here. Cross-paper comparisons are not like-for-like: teachers, tasks, cardinalities, and metrics differ across sources.

Five gaps limit what this survey can conclude. First, no reviewed source ablates KL direction with a student refit on a fixed-cardinality head, so the split-metric prediction is a mechanism awaiting a run. Second, all symmetric-loss evidence comes from generation or intermediate alignment [4][6][11], and the head-level transfer is UNVERIFIED. Third, the Prior Networks counterpoint concerns Dirichlet priors rather than softmax matching [7]. Fourth, DistillLens is recent (February 2026) with a placeholder-style code link, so it is one supporting point rather than a load-bearing result. Fifth, the Hugging Face model counts are fetch-time snapshots; they evidence availability of reverse-KL artifacts, not quality.

Open questions

Does symmetric KL preserve non-target tail ranking better than forward KL after the refit on a 77-way choice head at matched accuracy, measured by classwise-ECE and teacher-student tail correlation?
Does reverse KL lose non-target dark knowledge that the calibration refit cannot restore, or do forward, reverse, and symmetric students converge to the same post-refit calibration?
What skew interpolation between forward and reverse KL, if any, maximizes post-refit classwise calibration per unit of distillation compute on high-cardinality heads?

References

← All papers