Where Refusal Should Be Learned in the Training Stack

This paper asks where in the training stack refusal should be learned, and how each possible placement interacts with calibration trained through proper scoring rules. It builds on the companion survey of abstention mechanisms and scoring, which compared triggers, thresholds, and metrics, and on the proposal for reinforcement learning (RL) against proper scoring rules, which derives training objectives that reward honest probabilities. It does not repeat their derivations or benchmark tables. All arXiv IDs below were verified against their abstract pages, and every number is attributed inline to a source fetched for this paper. Velofy has trained no model for this paper and ships none with it.

The question

A typed-decision model takes state plus a declared question and returns a label, a score, or a boolean probability, usually with a confidence value. Refusal is the behavior of declining to answer: returning an explicit abstain signal instead of a decision. The parent paper surveyed how refusal can be triggered and scored. This paper asks a placement question: at which point in the stack should the refusal behavior be learned?

There are four candidate locations, and they are not mutually exclusive, but each additional one costs capacity, data, or complexity:

1. Pretraining signal: the base model arrives with representations that already distinguish known from unknown inputs, so refusal reads off something pretraining built.
2. Refusal-aware tuning data: the model is tuned on data that demonstrates refusal on unanswerable items, so refusal is a learned response pattern.
3. Selection head: a learned module, trained jointly with the decision head, decides per input whether to answer or abstain.
4. Post-hoc threshold: no refusal is learned at all; a cutoff fitted on held-out data gates a calibrated confidence at deployment time.

The second half of the question is the interaction with proper-scoring-rule calibration. A strictly proper scoring rule is a loss whose expected score is maximized only when the model reports its true probabilities, as Shao et al. define it [12]. Training against such rules (log score, Brier score, spherical score) is the mechanism by which models like Laya and the proposed Maestro Flash aim to produce honest confidences. But refusal changes what honesty means: a model that can abstain has a different output space, a different coverage, and a different conditional accuracy than one that must answer everything. The placement of refusal determines whether calibration and refusal are learned jointly under one objective or bolted together across stages that may disagree.

Background

Each location needs a one-paragraph definition so the evidence can be sorted.

Pretraining signal means refusal-relevant information present in the base representations before any refusal-specific training. The test is whether a probe, a secondary head, or the model’s own self-evaluation can predict answerability from pretrained features, and whether that prediction is calibrated. This is the cheapest location if it works, because it spends no refusal-specific training, but it is also the least controllable: pretraining optimizes for something else, and whatever uncertainty signal survives is incidental.

Refusal-aware tuning data means supervised or instruction-tuning examples in which the correct output on selected inputs is a refusal. The key design choice is how those inputs are selected: by human labeling, by adversarial construction (as in SQuAD 2.0 style unanswerable items, see the unanswerable-items paper), or by intersecting tuning questions with the model’s own parametric knowledge. Tuning changes weights, so it can reshape both the decision boundary and its calibration, for better or worse.

Selection head means a learned abstention module trained jointly with the predictor. The two main designs are a separate selection function optimized alongside classification, and an extra abstain class folded into the output distribution. Both move the coverage decision into training: the operating point becomes a training hyperparameter rather than a deployment knob.

Post-hoc threshold means fitting a cutoff (and usually a calibration map first) on held-out data, then gating at serving time. The official scikit-learn documentation describes the standard form: a calibrator, either a sigmoid (Platt) or isotonic regressor, is fit on held-out classifier outputs, mapping raw scores into probabilities [15]. Temperature scaling is the neural-network analogue, a single-parameter variant of Platt scaling that Guo et al. report as surprisingly effective on most datasets [5]. Nothing about refusal is in the weights; everything is in the fitted cutoff.

Proper-scoring-rule training cuts across all four. The log score (cross-entropy) is strictly proper but, as Shao et al. note, it is the only local scoring rule, depending solely on the probability of the observed outcome [12]. Non-local rules such as Brier and spherical scores consider the full distribution, which is why they can reward different calibration behavior. Laya’s README states that it is trained with reinforcement learning against strictly proper scoring rules, serves typed decisions over 100-plus languages in a single forward pass at 33 ms (7.2 ms per question batched) [16], and its public checkpoint card is convaiinnovations/laya, an Apache-2.0 text-classification model [17]. The Laya teardown examines what that training produces; the question here is where refusal fits around it.

What the evidence says

Pretraining signal: present but uncalibrated. Kadavath et al. train models to predict P(IK), the probability that they know the answer to a question without reference to any particular answer [8]. They report that models perform well at predicting P(IK) and partially generalize across tasks, but struggle with calibration of P(IK) on new tasks. Predicted P(IK) also rises appropriately with relevant source material or hints in context, which shows the signal tracks genuine answerability rather than surface features. Lin et al. find that GPT-3 can learn to express calibrated uncertainty in words, remaining moderately calibrated under distribution shift, and they provide evidence that this generalization depends on pretrained latent representations that correlate with epistemic uncertainty over answers [9]. Taken together, these results say pretraining builds representations that carry answerability information, but that information is not by itself a calibrated refusal trigger. None of the surveyed sources reports refusal emerging from pretraining alone without refusal-specific tuning or thresholding.

Refusal-aware tuning data: effective and fragile. Zhang et al. introduce R-Tuning, which constructs refusal-aware data from the intersection of pretrained knowledge and tuning data, then tunes the model to refuse questions beyond its knowledge [6]. They report that refusal generalizes to out-of-domain datasets as a meta-skill, and that learning uncertainty this way yields better calibration and better uncertainty estimation than uncertainty-based testing. The method’s repository notes it received an Outstanding Paper Award at the 2024 conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2024) [7]. The counterpoint comes from AbstentionBench, which evaluates 20 frontier large language models (LLMs) across 20 datasets and reports that abstention is unsolved, that scaling is of little use, and that reasoning fine-tuning degrades abstention by 24% on average, even in the math and science domains the reasoning models are trained for [14]. Its documentation adds human-validated judges for scalable evaluation across 6 abstention scenarios [14]. So tuning data can install refusal as a generalizing skill, but the tuning objective dominates: accuracy-only or reasoning-only tuning actively uninstalls it. A system prompt helps in practice without resolving the underlying inability, per the same benchmark.

Selection head: the best tradeoff at the highest integration cost. SelectiveNet trains classification and rejection jointly, end to end, and its authors report a consistently improved risk-coverage tradeoff over several classification and regression datasets, reaching new state-of-the-art results for deep selective classification [2]. Deep Gamblers takes the alternate route of folding abstention into the output space: the m-class problem becomes m+1 classes with the extra class meaning abstain, trained with a loss based on the doubling rate of gambling, and the authors report strong results on SVHN and CIFAR10 at various coverages with almost no modification to the inference algorithm or architecture [3]. The post-hoc contrast is the earlier selective-classification method, which takes a trained network as given and rejects at test time to meet a desired risk level, reporting 2% error on ImageNet top-5 with probability 99.9% at almost 60% test coverage [1]. The pattern is consistent: learning the reject option jointly beats thresholding a fixed network’s confidence on the risk-coverage tradeoff, but the operating point moves into training, and the model must be retrained if application costs change substantially.

Post-hoc threshold: the cheapest floor, and it inherits every upstream sin. Hendrycks and Gimpel establish the baseline: correctly classified examples tend to have greater maximum softmax probabilities than misclassified and out-of-distribution examples, and thresholding that probability works across vision, language, and speech tasks [4]. Guo et al. show modern networks are poorly calibrated, identify depth, width, weight decay, and batch normalization as influencing factors, and report temperature scaling as surprisingly effective on most datasets [5]. The limit appears under shift: Ovadia et al. benchmark calibration methods under dataset shift at scale and find that traditional post-hoc calibration falls short, while some methods that marginalize over models give surprisingly strong results across tasks [10]. Post-hoc is therefore the right default and the right floor to beat, but its refusal quality is exactly its calibration quality on the deployed distribution, no better.

Analysis

The table sorts mechanisms by location. The interaction question is which locations compose with proper-scoring-rule calibration and which fight it. Four findings emerge.

First, the training loss sets the calibration floor that every later refusal stage inherits. Guo et al. document that standard cross-entropy training leaves modern networks systematically overconfident [5]. Mukhoti et al. show the loss choice alone moves that floor: focal loss learns models that are already very well calibrated, unlike standard cross-entropy, and combined with temperature scaling yields state-of-the-art calibrated models while preserving accuracy, across computer vision and natural language processing (NLP) datasets and many architectures [11]. Shao et al. go further, substituting Brier and spherical scores for the log score in language generation and reporting substantial improvements without adjusting other hyperparameters, scaling up to LLaMA-7B and LLaMA-13B [12]. For refusal placement, the lesson is that post-hoc thresholding on top of a miscalibrated base is damage control, while proper-scoring-rule training (or a calibration-friendly loss) reduces the correction any later stage must apply. This is the premise of the companion RL proposal: train honesty in rather than fitting it afterward.

Second, tuning-stage refusal composes with calibration when the tuning objective is itself a proper scoring rule, and fights it otherwise. The cleanest demonstration is Reinforcement Learning with Calibration Rewards (RLCR): ordinary binary-reward RL degrades calibration and increases hallucination in other domains, while augmenting the reward with a Brier score term yields models the authors prove (for any bounded proper scoring rule) to be both accurate and well calibrated, with substantially improved calibration at no accuracy loss in and out of domain, outperforming both ordinary RL and post-hoc confidence classifiers [13]. R-Tuning corroborates from the data side: learning refusal from knowledge-intersection data improves calibration beyond uncertainty-based testing [6]. AbstentionBench corroborates from the failure side: reasoning fine-tuning with accuracy-shaped rewards degrades abstention 24% on average [14]. The three results triangulate one rule: if the tuning stage prices uncertainty honestly, refusal tuning and calibration reinforce each other; if it prices only correctness, tuning buys accuracy with calibration and refusal both degrading. Placement at the tuning stage is therefore not one choice but two, and the objective matters more than the data format.

Third, the selection head is the only location that learns refusal probabilities under the same proper scoring rule as decision probabilities, which is both its strength and its coupling risk. In the Deep Gamblers framing, abstention is class m+1 of the same softmax, so a strictly proper rule over that extended distribution rewards honest abstain probabilities jointly with honest label probabilities [3]. In SelectiveNet, the joint optimization targets the covered domain directly [2]. The elegance is real: there is no separate trigger whose calibration must be validated independently. But the coupling means the price of abstention versus answering is fixed at training time. A deployment whose cost ratio differs from the training assumption cannot simply move a threshold; the post-hoc escape hatch (thresholding the learned abstain probability) reintroduces exactly the stage the joint design was meant to subsume. For small decision models with per-bucket operating points (question type, option count, language), this rigidity argues for keeping at least a post-hoc gate even when a selection head exists, so each bucket can be re-pointed without retraining.

Fourth, post-hoc thresholding is the correct first refusal stage for any proper-scoring-rule trained model, because it preserves the separation between honest probabilities and cost-dependent decisions. Proper scoring rules train the model to report true beliefs; the decision to act on a belief at stated costs is a downstream computation that should not distort the belief itself. Fitting per-bucket temperatures and then per-bucket thresholds keeps those layers honest: the temperature says what the model believes, the threshold says what the application can afford. Ovadia et al. bound this claim: under distribution shift, post-hoc calibration falls short [10], so the threshold layer needs shift-aware validation (held-out slices that resemble deployment, not just a random split) and cannot be the only layer where shift is handled. The practical stack that follows is: proper-scoring-rule training as the base, refusal-aware tuning data to install the abstain behavior in-schema, per-bucket post-hoc calibration plus thresholds as the deployment gate, and a selection head only if measurements show the post-hoc ceiling binding on the slices that matter. Each layer is independently ablatable on the same backbone, which is what makes the placement question experimentally tractable.

One measurement corollary follows. Calibration for a refusing model must be selective calibration: expected calibration error (ECE) computed on answered items at the deployed operating point, alongside the risk-coverage curve, not a single global ECE number. A model can improve global ECE by refusing its worst-calibrated slice while its answered-subset calibration stays flat, or degrade coverage while looking better calibrated. Any placement comparison that reports only accuracy and global ECE confounds the refusal effect with the calibration effect. The sigmoid-calibration paper in this graph makes the analogous point for per-bucket refits; the same discipline applies here.

Implications for small decision models

For Jev, the placement question is currently unanswerable from public evidence, and that constrains what callers can do. Jev is a closed application programming interface (API), as the Jev survey documents, so where its refusal behavior is learned, if anywhere, is UNVERIFIED. The independent decision-model benchmark (DMB) reports Jev at 76.3% on 77-way banking intent, p50 latency 264 to 276 ms flat from 2 to 255 options, a sharp 255-choice cap, a 13% option-permutation flip rate, ECE 0.246 against an LLM range of 0.039 to 0.122, and ignorance admitted on 49.7% of forced-uncertainty items where every tested LLM admits 97.3 to 100% [18]. The implication for placement is narrow: callers cannot tune Jev’s data or head, so their only refusal stage is post-hoc gating on returned confidences, and DMB’s ECE and forced-uncertainty numbers say that gate is unvalidated on exactly the inputs where refusal matters. Until a future API revision exposes refusal placement (or at least an explicit abstain output), Jev should be treated as a model with post-hoc-only refusal available to its users, with the calibration caveats above.

For Laya, the path is incremental and each step is measurable. Laya already does proper-scoring-rule RL per its README, serves a single forward pass at 33 ms, and publishes Apache-2.0 checkpoints [16][17]; its BENCHMARKS.md cautions that its Jev comparison figures are third-party published and never measured against a live API, and that older calibration columns predate the temperature clamp [19]. The cheapest placement experiment adds per-bucket abstention thresholds fitted on held-out data next to the existing per-bucket temperatures, pure post-hoc, no retraining. The next step adds refusal-aware tuning data in the R-Tuning style, labeling what the encoder knows via its own held-out accuracy per slice rather than LLM self-consistency, and checks whether selective calibration improves as R-Tuning reports [6] or degrades as accuracy-only tuning would predict [14]. The selection head comes last: a SelectiveNet-style joint objective or a Deep Gamblers-style abstain class over the option set, justified only if it beats the post-hoc risk-coverage curve on the same backbone. The teardown’s documented failure regions (saturated confidence where accuracy is zero) are the slices where each stage must prove itself; see the Laya teardown.

For Velofy’s proposed Maestro Flash, placement should be decided by experiment design, not by default. Maestro Flash has no weights and no results, so its contribution at this stage is a measurement plan that keeps every location ablatable. Four design choices follow from the analysis. First, fix the schema decision early: abstain as an explicit in-schema output with reason codes (tuning-data and head compatible) versus caller-side thresholding on confidences (post-hoc only), since the schema determines which placements are even expressible. Second, train all heads against proper scoring rules from the start, per the companion proposal, so the calibration floor is honest before any refusal stage is added. Third, add an unanswerable slice constructed to be exactly gradable without a judge, following the construction paper, scored with abstention precision and recall plus selective ECE at fixed coverage. Fourth, run the placement ablation the field lacks at small-encoder scale: same backbone, same unanswerable slice, four configurations (post-hoc only, plus tuning data, plus selection head, all combined), reporting full risk-coverage curves rather than single operating points. If the post-hoc configuration matches the learned ones on the slices that matter, the learned stages are complexity without benefit; if a learned stage wins, the curve shows where and at what coverage cost.

Limitations

This survey ran no experiments; every empirical claim is attributed to its source, and the interaction analysis is synthesis across papers with different models, tasks, and setups, not a controlled ablation. No surveyed paper compares all four placements on one backbone, so the recommended stack order (proper-scoring-rule base, tuning data, post-hoc gate, head only if needed) is an inference from convergent results, not a measured ranking. Most refusal evidence comes from vision classifiers with hundreds of classes or generative LLMs with open-ended outputs; small encoders with closed typed outputs remain the gap in the middle, and only behavioral third-party measurements (DMB on Jev, Laya’s self-reported benchmarks) touch that regime. Numbers drawn from different benchmarks are not directly comparable: a 24% abstention degradation on AbstentionBench and a risk-coverage curve on CIFAR do not share units. Finally, every operating point depends on application-specific costs of wrong answers versus abstentions, so no universal placement or threshold can be recommended; each deployment must fit its own point on its own held-out data.

Open questions

1. Does adding refusal-aware tuning data to a proper-scoring-rule trained small encoder preserve, improve, or degrade selective calibration at fixed coverage, measured against a post-hoc-only baseline on the same backbone and unanswerable slice?
2. What is the smallest selection-head or abstain-class design that beats per-bucket post-hoc thresholding on risk-coverage for a small typed-decision encoder, and does the winning design change the optimal temperature grouping?
3. How should selective calibration (ECE on answered items at the deployed operating point) be reported alongside refusal metrics so that a change in application costs does not require rerunning the evaluation?

References

1. Geifman and El-Yaniv, Selective Classification for Deep Neural Networks. arXiv:1705.08500
2. Geifman and El-Yaniv, SelectiveNet: A Deep Neural Network with an Integrated Reject Option. arXiv:1901.09192
3. Liu et al., Deep Gamblers: Learning to Abstain with Portfolio Theory. arXiv:1907.00208
4. Hendrycks and Gimpel, A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. arXiv:1610.02136
5. Guo et al., On Calibration of Modern Neural Networks. arXiv:1706.04599
6. Zhang et al., R-Tuning: Instructing Large Language Models to Say “I Don’t Know”. arXiv:2311.09677
7. R-Tuning official repository (method summary and NAACL 2024 Outstanding Paper Award note). https://github.com/shizhediao/R-Tuning
8. Kadavath et al., Language Models (Mostly) Know What They Know. arXiv:2207.05221
9. Lin et al., Teaching Models to Express Their Uncertainty in Words. arXiv:2205.14334
10. Ovadia et al., Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. arXiv:1906.02530
11. Mukhoti et al., Calibrating Deep Neural Networks using Focal Loss. arXiv:2002.09437
12. Shao et al., Language Generation with Strictly Proper Scoring Rules. arXiv:2405.18906
13. RLCR: Beyond Binary Rewards, Training LMs to Reason About Their Uncertainty. arXiv:2507.16806
14. Kirichenko et al., AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions, and its documentation repository. arXiv:2506.09038 and https://github.com/facebookresearch/AbstentionBench
15. scikit-learn user guide, Probability calibration (official documentation). https://scikit-learn.org/stable/modules/calibration.html
16. NandhaKishorM/laya README (architecture, RLCD training, latency figures). https://github.com/NandhaKishorM/laya
17. convaiinnovations/laya model card on HuggingFace (license, pipeline tag). https://huggingface.co/convaiinnovations/laya
18. nibzard/decision-model-benchmark, independent typed-decision benchmark README. https://github.com/nibzard/decision-model-benchmark
19. NandhaKishorM/laya BENCHMARKS.md (self-reported figures and provenance notes). https://github.com/NandhaKishorM/laya/blob/main/BENCHMARKS.md

← All papers