Rationale Supervision Versus Soft Targets

This paper is a child of the distillation survey, which left the rationale-supervision question UNVERIFIED. It builds on that survey and on the teacher temperature and Kullback-Leibler (KL) direction survey without repeating their method tables or benchmark summaries. All arXiv IDs below were verified against their abs pages, every number is attributed inline to a source fetched for this paper, and Velofy has trained no model for this paper and ships none with it.

The question

Does rationale-derived supervision through filtering, loss weighting, or an auxiliary head beat plain soft-target distillation for single-pass encoder decision heads at a matched teacher-query budget?

The question names three mechanisms, one student architecture, and one budget constraint. The mechanisms are data filtering (keep only examples whose teacher rationale supports the label), loss weighting (scale each example’s loss by a rationale-consistency score), and an auxiliary head (train the encoder to predict something rationale-derived alongside the decision, then discard that head at inference). The student is a single-pass encoder with typed decision heads in the style of Jev, Laya, or the proposed Maestro Flash: state and options go in, a probability vector comes out, no tokens are emitted. The budget constraint is matched teacher-query spend: rationale queries cost extra output tokens per example, so any comparison that ignores query cost compares a cheap large dataset against an expensive small one and learns nothing.

The question matters because the rationale-distillation literature reports large gains, but every one of those gains was measured on a student that generates text. The distillation survey on this site flagged the structural mismatch: a typed-decision encoder has nowhere for a rationale to go at inference time, so the only viable adaptations are indirect. This survey asks whether the indirect adaptations have any evidence behind them.

First principles

A single-pass encoder head is trained by one loss over a probability vector. That fixes where rationale information can enter and what it can cost.

The loss has only a few inputs teacher output can touch. It can set the target for each example, decide which examples are present, set how much each one counts, or add extra targets for the shared encoder. Soft-target distillation works through the first. Filtering, loss weighting, and an auxiliary head work through the other three.

At inference, a rationale has nowhere to go. The head emits a probability vector and no tokens. Whatever a rationale contributes must already be in the weights when training ends.

Rationales cost tokens, and tokens cost examples. At a fixed teacher budget, the number of training examples is the budget divided by the cost per example. A label-only query pays for the input and a short answer. A rationale query adds the rationale’s output tokens, and sampling several rationales to check agreement multiplies that. At equal spend, every rationale arm trains on fewer examples than the soft-target control.

A rationale is evidence about the teacher, not about the truth. It helps as a filter or a weight only to the extent that its quality tracks label correctness. When explanations are unfaithful, as the counter-evidence below shows, fluent text and correct labels come apart.

Correctness signals do not constrain probabilities. Every rationale mechanism rewards right labels or good steps. None penalizes an overconfident probability, so calibration has to be measured separately, after the same refit, on every arm.

Background

The soft-target baseline is defined in the distillation survey: the student matches the teacher’s softened output distribution over the decision space, which needs only a distribution per example and survives teachers reachable only through an application programming interface (API). That survey also establishes the calibration discipline this node inherits: accuracy transfer and calibration transfer are separate, the teacher temperature and the student temperature are different objects fit separately, and any distillation claim must report Brier score and expected calibration error (ECE) after a held-out refit at matched accuracy. What follows does not re-argue those points.

The rationale literature starts from chain-of-thought (CoT) prompting. Wei and coauthors show that generating intermediate reasoning steps improves large language model (LLM) performance on arithmetic, commonsense, and symbolic reasoning tasks, and report that prompting a 540B parameter model with eight chain-of-thought exemplars reaches state of the art accuracy on Grade School Math 8K (GSM8K), surpassing even finetuned GPT-3 with a verifier [1]. The premise it created matters here: the reasoning trace carries signal beyond the final answer, so the trace itself can be training data.

Two framings of that signal shape the three mechanisms in the question. Lightman and coauthors distinguish outcome supervision, which gives feedback on a final result, from process supervision, which gives feedback on each intermediate reasoning step [3]. Filtering and loss weighting are outcome-supervision ideas (use rationale quality to select or weight whole examples), while an auxiliary head is closer to process supervision (supervise intermediate representations, not just the final label). The second framing comes from the student side. The Hugging Face Transformers documentation describes T5 (Text-to-Text Transfer Transformer) as an encoder-decoder transformer, available from 60M to 11B parameters, that treats every natural language processing (NLP) task as text generation with task-specific prefixes [12]. For such a student, predicting a rationale is just another text target, so rationale multitask training slots into the existing architecture. A single-pass encoder head emits no tokens, which is why the direct recipe cannot transfer and the question asks only about the indirect ones.

The nearest classifier-shaped precedent is e-SNLI. Camburu and coauthors extend the Stanford Natural Language Inference (SNLI) dataset with human-annotated natural language explanations of the entailment relations, train models that incorporate the explanations during training and output them at test time, and report uses including full sentence justifications, improved sentence representations, and transfer to out-of-domain natural language inference (NLI) datasets [7]. NLI is a three-way decision, so this is closer to typed decisions than math word problems are. But the students output explanations at test time, which keeps them on the generative side of the line this node draws.

What the evidence says

The positive evidence for rationale supervision is strong, consistent, and uniformly generative. Five papers report large gains from training smaller models on teacher rationales, all with students that emit text.

Hsieh and coauthors introduce distilling step-by-step, which extracts LLM rationales as additional supervision for small models in a multi-task framework [2]. Across four NLP benchmarks they report three findings: better performance than finetuning or distillation with far fewer training examples, better performance than few-shot prompted LLMs with far smaller models, and a headline result in which a finetuned 770M T5 outperforms a few-shot prompted 540B PaLM using only 80 percent of available data on one benchmark, where standard finetuning of the same T5 cannot match the teacher even with the full dataset [2]. This is the result most often cited for rationale supervision, and its student is a text-to-text model predicting labels and rationales as token sequences.

Zelikman and coauthors propose STaR (Self-Taught Reasoner), a bootstrapping loop that generates rationales for many questions, retries with the correct answer supplied when the first attempt is wrong, finetunes on the rationales that yielded correct answers, and repeats [5]. They report significant improvements over finetuning to predict answers directly, and performance comparable to finetuning a 30 times larger state-of-the-art language model on CommonsenseQA [5]. STaR is filtering in its purest form, but its filter needs ground-truth answers, which bounds where the recipe applies.

Lightman and coauthors compare process supervision against outcome supervision for training models on the MATH dataset, and report that process supervision significantly outperforms outcome supervision, with their process-supervised model solving 78 percent of problems from a representative MATH test subset [3]. They also report that active learning significantly improves the efficacy of process supervision, and release PRM800K, 800,000 step-level human feedback labels whose README describes two-phase labeling with quality control questions served to every labeler [3][10]. This is the loss-weighting analogue at full strength: per-step quality scores steering training. Its cost is human labeling at a scale no decision-distillation budget contemplates, which is why the cheap version (LLM-judged step quality) is the one that needs testing.

Shridhar and coauthors distill step-by-step reasoning into smaller models with Socratic CoT, which decomposes each problem into subproblems and trains two small models (a decomposer and a solver) that work in sync [8]. They report gains over 70 percent relative to baselines on GSM8K, StrategyQA, and SVAMP, and cases where a GPT-2 large student outperforms a 10 times larger GPT-3 6B model [8]. Mukherjee and coauthors train Orca, a 13B model, on explanation traces and step-by-step thought processes from GPT-4 with teacher assistance from ChatGPT, and report surpassing Vicuna-13B by more than 100 percent on Big-Bench Hard (BBH) and 42 percent on AGIEval, reaching parity with ChatGPT on Big-Bench Hard [9].

A sixth result operates at data-collection time rather than training time. Wang and coauthors propose self-consistency: sample a diverse set of reasoning paths, then select the most consistent answer by marginalizing out the paths [6]. They report gains over chain-of-thought prompting of 17.9 points on GSM8K, 11.0 on SVAMP, 12.2 on AQuA, 6.4 on StrategyQA, and 3.9 on ARC-challenge [6]. For this node, self-consistency is a filtering and weighting scheme in disguise: agreement across sampled paths is a per-example quality score that needs no ground truth and no human labels, only repeated teacher queries. Its cost lands directly on the matched-budget constraint, since every extra sample is extra spend.

Adoption evidence shows explanation supervision folded into large-scale training. The Hugging Face model API lists esnli among the dataset tags of google/flan-t5-base, which it reports at 1,299,490 downloads and 1093 likes at the time of writing, and the esnli/esnli dataset card exists with 1727 downloads and 23 likes [11][13].

The counter-evidence comes from faithfulness research. Turpin and coauthors show that chain-of-thought explanations systematically misrepresent the reasons for model predictions: adding biasing features (for example, reordering multiple-choice options so the answer is always “(A)”) shifts predictions without any mention in the explanations, and models frequently rationalize answers they were biased toward [4]. They report accuracy drops of as much as 36 percent on 13 Big-Bench Hard tasks with GPT-3.5 and Claude 1.0, plus social-bias cases where explanations justify stereotyped answers without mentioning the bias [4]. Rationale presence therefore does not certify label quality: a filter that keeps examples for fluent rationales selects for confident confabulation. The option-reordering bias is also the generative analogue of the option-order sensitivity this site’s permutation sensitivity survey studies in encoders.

No reviewed source reports calibration: no ECE, no Brier score, no reliability diagram. Rationale supervision is evaluated purely on correctness, so whether it transfers, improves, or harms calibration is UNVERIFIED in every source reviewed here.

Method	Student type	Rationale mechanism	Reported result	Single-pass encoder transfer
Distilling step-by-step [2]	770M T5, generative	Multi-task label plus rationale	Beats few-shot 540B PaLM on 80 percent of data	No direct form; motivates the auxiliary head arm
STaR [5]	Generative language model (LM)	Correctness filtering, iterated	Matches finetuning a 30x larger LM on CommonsenseQA	Filtering form applies; needs ground-truth answers
Process supervision [3][10]	Generative LM plus reward model	Per-step quality labels	78 percent on a MATH subset; beats outcome supervision	Weighting analogue needs step labels; human cost is 800k labels
Socratic CoT distillation [8]	Two small generative models	Decomposition plus solving	Over 70 percent gains on GSM8K, StrategyQA, SVAMP	No direct form; decomposition needs generation
Orca [9]	13B generative	Explanation-trace imitation	Over 100 percent over Vicuna-13B on BBH; ChatGPT parity	No direct form; traces need a generative student
Self-consistency [6]	Generative LM at decode time	Agreement across sampled paths	Up to 17.9 points on GSM8K	Filtering and weighting form applies; costs repeated queries
e-SNLI training [7]	Models emitting explanations	Human explanations in training	Justifications, better representations, out-of-domain (OOD) transfer	Nearest precedent; test-time output still generative
Unfaithful CoT [4]	GPT-3.5, Claude 1.0 (evaluated)	Counter-evidence on rationales	Up to 36 percent accuracy drop under biasing features	Constrains all three arms: filter on agreement, not fluency
Soft-target distillation	Any classifier	Teacher distribution matching	See the distillation survey	The control arm; needs only decision-space outputs

Analysis

Six conclusions follow, in decreasing order of confidence. The first is close to a deductive reading of the table. The rest are judgments about experiment design that the evidence motivates without settling.

First, the direct question is UNVERIFIED, and the table shows why. Every positive rationale result trains a student that emits tokens, and the two mechanisms that could apply to encoders without modification (STaR-style filtering and self-consistency agreement) were never tested on an encoder head against a soft-target control at a matched query budget. Anyone claiming rationale supervision beats soft targets for single-pass decision heads is extrapolating from generative students.

Second, filtering trades examples for quality at fixed spend, and the exchange rate is unknown. A rationale query costs the label query plus the rationale’s output tokens, and self-consistency filtering multiplies that by the sample count. At a fixed teacher budget, the filtered training set is therefore smaller than the unfiltered set the soft-target control can afford, potentially much smaller. STaR suggests cleaner data can win [5], but STaR never faced a budget-matched control with more examples. The honest comparison fixes total teacher spend in input plus output tokens and asks whether fewer filtered examples beat more unfiltered ones. The Turpin result [4] adds a design constraint: the filter should key on cross-sample agreement (self-consistency style), not on rationale presence or fluency, since fluency is cheap and bias is silent.

Third, loss weighting faces a label-cost problem that process supervision makes concrete. The 78 percent MATH result rested on 800,000 human step labels collected in two phases with quality control [3][10]. Nothing in a decision-distillation budget resembles that. The affordable version scores rationale consistency with the teacher itself (sample k traces, weight by agreement) or with a small judge model, but whether machine-judged consistency carries the process-supervision edge is UNVERIFIED, and the judge inherits the faithfulness problem: a judge that rationalizes is a weight corrupted at the source. Weighting is the weakest-motivated of the three arms until someone shows machine-judged weights beat uniform weights at matched spend.

Fourth, the auxiliary head is the only arm that changes the student rather than the data, and it has the thinnest precedent. The concrete form is a head on the pooled encoder representation that predicts something rationale-derived (an entailment score between rationale and chosen option, a bag-of-reasons embedding, or the teacher rationale’s sentence embedding), trained jointly with the decision heads and discarded at inference. e-SNLI is the nearest motivation, but its models output explanations at test time [7], so it does not test whether rationale prediction as a pure training auxiliary improves decisions. The mechanism is plausible (the auxiliary loss shapes the shared encoder toward rationale-relevant features) and cheap at inference (the head is discarded), but whether it helps, does nothing, or steals capacity from the decision heads is UNVERIFIED.

Fifth, calibration is unmeasured on every rationale arm, so the refit discipline from the sibling surveys binds here unchanged. Rationale supervision optimizes correctness signals (right label, consistent trace, good steps), and nothing in those objectives promises honest probabilities. The teacher temperature survey argues that Brier comparisons are confounded unless accuracy is matched, and the same applies to any rationale-versus-soft-target claim: report accuracy, Brier score, and ECE after the identical held-out refit, and compare within accuracy bands. If a rationale arm wins on accuracy but loses on post-refit ECE, that is a finding about what rationale supervision transfers, not a failure of the experiment.

Sixth, the full comparison is a four-arm study with cached teacher outputs, and its run count is affordable. The arms are plain soft targets (control), agreement-filtered labels, agreement-weighted losses, and soft targets plus an auxiliary rationale head, all drawn from one cached teacher-output pool so the query budget is identical by construction. The pool must record per-example input and output token counts so spend is auditable. A finetune-only baseline (no teacher signal) belongs in the design as the floor, following the baseline discipline of the distillation survey. High-cardinality accuracy must be reported separately from low-cardinality accuracy, since the failure modes differ and averaging hides them. The interesting outcome is not a single winner but the budget curve: which arm wins at 10k, 100k, and 1M teacher queries may differ, and the crossover points are the deployment-relevant result.

Implications for small decision models

For Jev, only the filtering arm is reachable from outside, and only in its expensive form. Jev is a closed API whose public framing returns typed probabilistic decisions; the Jev survey gives the full behavioral picture. Whether its API exposes anything rationale-like is UNVERIFIED here, so the available moves are response-level: query each example multiple times, keep high-agreement decisions, and train on the filtered set. That is self-consistency filtering with the sampling budget paid per example at API prices: every agreement check is a query not spent on a new example. The weighting and auxiliary-head arms need rationale text the API does not advertise, so they are out of reach for a Jev teacher until that changes.

For Laya, the experiment is fully runnable but the teacher role needs care. Laya is open, so its logits and full distributions are accessible, and its BENCHMARKS.md headline table (fetched for this paper) reports typed-decisions 0.766 against Jev at 0.727, AG News 0.953 against 0.910, DAIR Emotion 0.600 against 0.480, ECE after temperature fitting 0.081 against 0.246, and median (p50) latency of 32.8ms on an NVIDIA T4 graphics card against 236 to 276ms, with the file’s own warning that its Jev figures are third-party published and never measured in that repository [14]. Laya’s per-bucket temperature practice is the closest decision-model precedent for the student-side refit this node’s comparisons require. But Laya emits no rationales, so it cannot be the rationale teacher; the natural design uses an open LLM as teacher and Laya’s architecture as the student, with high-cardinality results reported separately given the option-encoding bottleneck the Laya teardown documents.

For the proposed Maestro Flash, which has no weights and no results, the implication is a concrete recipe for testing the rationale question inside its distillation stage. One, use an open LLM teacher that emits both decisions and rationale traces, and cache one output pool (labels, traces, multiple samples per example, token counts) reused across all arms. Two, run the four-arm comparison above (soft-target control, filtered, weighted, auxiliary head) plus a finetune-only floor, at two or three total-budget points to trace the budget curve. Three, key every rationale-derived signal on cross-sample agreement rather than single-trace fluency, following the Turpin constraint [4]. Four, refit per option-count bucket with clamping after every arm, and report accuracy, Brier score, and ECE on shared held-out data with high-cardinality splits separated. Five, treat a null result (no arm beats soft targets at matched spend) as a publishable finding that simplifies the training stack, not as a failed experiment.

Limitations

This survey read abstracts, repository files, model and dataset APIs, and official documentation, not full paper texts or codebases, and it ran no experiments. Every quantitative claim is attributed inline to the source that reported it, and those figures are self-reported by their authors under their own protocols, not independently reproduced here. Cross-paper comparisons are not like-for-like: the teachers, students, tasks, and metrics differ across sources, and a 70 percent gain on GSM8K [8] cannot be weighed against a 36 percent drop on Big-Bench Hard [4].

Three specific gaps limit what this survey can conclude. First, the encoder-head configuration this node asks about appears in no reviewed source as a reported experiment; the transfer column of the table is analysis, not evidence. Second, teacher-query costs are modeled, not measured: token counts per rationale and per sample are asserted as the right budget unit but no reviewed source prices them. Third, the auxiliary-head arm has no direct precedent anywhere in the reviewed literature, so its design sketch rests on the e-SNLI analogy [7] rather than on any tested instance.

Open questions

At a fixed teacher-query spend measured in input plus output tokens, does agreement-filtered teacher supervision beat plain soft-target distillation on a larger unfiltered set for a single-pass choice head, judged on accuracy and post-refit ECE?
Does an auxiliary rationale-prediction head discarded at inference improve decision accuracy or post-refit calibration over soft targets alone at matched data, and which rationale representation transfers best?
Do rationale-consistency loss weights transfer teacher calibration or only accuracy, and does a per-bucket temperature refit erase any difference between weighted and unweighted students?

References

← All papers