Abstention and Honest Uncertainty for Small Decision Models

This paper is a survey of abstention: when a model should decline to answer, what form that refusal should take, and how to score it. It builds on two earlier papers in this graph, the Jev survey and the proper-scoring-rule reinforcement learning (RL) proposal, and does not repeat their derivations or benchmark tables. All arXiv IDs below were verified against their abs pages, and every number is attributed inline to a source fetched for this paper. Velofy has trained no model for this paper and ships none with it.

The question

A typed-decision model takes state and a declared question, then returns a label, a score, or a boolean probability, usually with a confidence value. The abstention question is what the model should do when it does not know: which inputs should trigger a refusal, what the refusal should look like on the wire, and how an evaluator should reward a correct refusal and punish a wrong answer or a needless refusal.

Abstention is distinct from low confidence, though the two are often conflated. Low confidence means the model answers and reports doubt, leaving the caller to gate on a threshold. Abstention means the model declines to answer at all, returning something like an explicit abstain flag instead of a decision. Both are forms of honest uncertainty, and a system can offer both, but they need different scoring. A confidence score is judged by calibration (does 70% mean 70%?) while an abstention is judged as a decision with its own precision and recall (did it refuse the unanswerable items and answer the answerable ones?).

The scoring problem is what makes this hard. A benchmark that only measures accuracy punishes honesty: a model that guesses on every unanswerable question can outscore one that correctly abstains. Abstention evals therefore need a cost model, explicit or implicit, in which a wrong answer costs more than an abstention, which costs more than a cheap correct answer.

First principles

Abstention is a decision made under costs. A few lines of arithmetic set its shape.

Every item ends in one of three outcomes. The model answers correctly, answers wrongly, or abstains. The application, not the model, sets what each outcome costs. A correct answer is cheapest. A wrong answer is most expensive. An abstention sits between them, because a person or a fallback system must then handle the item.

The costs set the threshold. Say a correct answer costs nothing, a wrong answer costs W, and an abstention costs A, with A smaller than W. If the model is right with probability p, answering costs (1 minus p) times W on average, and abstaining costs A. Answering is the better choice only when p is above 1 minus A/W. If a wrong answer costs ten times an abstention, the model should answer only when it is at least 90% sure.

The threshold is only as good as the probability. The rule takes p at face value. A model that says 0.95 but is right 70% of the time will answer items it should refuse. Calibration is therefore a precondition for abstention, not a separate feature.

Accuracy-only scoring makes guessing optimal. Scoring by accuracy alone charges an abstention the same as a wrong answer. With A equal to W, the threshold falls to zero and the best policy is to always answer. That is why benchmarks that ignore abstention punish honest models.

Not every refusal is uncertainty. A request can be rejected because it is malformed or over a limit, before the model judges anything. That is input validation, and it must be counted apart from decision-level abstention.

Background

The oldest framing is the reject option in classification, now usually called selective classification. The setup is a classifier paired with a selection function: the classifier predicts, the selector decides whether to output the prediction or abstain, and performance is measured as risk (error rate on answered items) against coverage (fraction of items answered). Geifman and El-Yaniv introduced this framing for deep networks, proposing a method that takes a trained network and a desired risk level, then rejects instances at test time as needed to meet that risk with high probability [1]. They report 2% error on ImageNet top-5 with probability 99.9% at almost 60% test coverage. The lesson is structural: abstention converts a fixed accuracy number into a tradeoff curve, and the operating point is a deployment choice, not a property of the weights.

A follow-up by the same authors, SelectiveNet, moved the reject option inside training, optimizing classification and rejection jointly instead of thresholding a pretrained network’s confidence [2]. The paper reports a consistently improved risk-coverage tradeoff on classification and regression datasets. The contrast matters for small models: post-hoc thresholding bolts onto any checkpoint, while an integrated reject option needs training support but spends capacity only where it answers.

A second lineage is out-of-distribution (OOD) and misclassification detection. Hendrycks and Gimpel showed that correctly classified examples tend to have greater maximum softmax probabilities than misclassified and out-of-distribution examples, and proposed thresholding that probability as a baseline across vision, language, and speech [3]. This is the simplest abstention trigger, one threshold on one number, and it fixes the floor every fancier method must beat.

A third lineage is calibration, covered in depth by the companion proper-scoring-rule paper. Guo et al. established that modern deep networks are systematically overconfident, and that temperature scaling, a single-parameter variant of Platt scaling, is surprisingly effective as a post-hoc fix on most datasets [4]. Calibration is a prerequisite for threshold-based abstention: if confidence does not mean what it says, no threshold on it can implement a desired risk level. This paper takes calibrated confidences as an input and asks what to do with them, including when to refuse outright.

A fourth lineage comes from question answering. The Stanford Question Answering Dataset (SQuAD) 2.0 combined the 100,000 questions of SQuAD 1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones, and required systems to abstain when no answer is supported by the paragraph [5]. The paper reports that a strong neural system reaching 86% F1 on SQuAD 1.1 achieves only 66% F1 on SQuAD 2.0. The official SQuAD site states the same requirement operationally: systems must determine when no answer is supported and abstain [6]. Unanswerability here is defined by construction rather than by a judge’s opinion, which is what makes SQuAD 2.0 the cleanest precedent for scoring abstention.

The fifth lineage is honesty in large language models (LLMs), which splits into three approaches. Kadavath et al. ask whether models can evaluate their own claims, using the probability P(True) that a proposed answer is correct and training models to predict P(IK), the probability that “I know” the answer without reference to any particular answer [7]. They report encouraging calibration and scaling for P(True), good P(IK) prediction with partial cross-task generalization, but poor P(IK) calibration on new tasks. Lin et al. show that GPT-3 can learn to express uncertainty in words, with stated confidence levels mapping to well-calibrated probabilities that remain moderately calibrated under distribution shift, introducing the CalibratedMath suite to test this [8]. Zhang et al., in R-Tuning, take the refusal route: they construct refusal-aware training data from the intersection of pretrained knowledge and tuning data, and tune the model to refuse questions beyond its knowledge [9]. They report that refusal generalizes out of domain as a meta-skill and that learning uncertainty this way improves calibration beyond uncertainty-based testing. The R-Tuning repository notes the paper received an Outstanding Paper Award at the 2024 conference of the North American Chapter of the Association for Computational Linguistics (NAACL) [10].

The sixth lineage is conformal prediction, which offers what the others do not: finite-sample guarantees. Angelopoulos and Bates introduce conformal prediction as a paradigm that wraps any pretrained model to produce sets guaranteed to contain the ground truth with a user-specified probability such as 90%, valid in a distribution-free sense with explicit non-asymptotic guarantees [11]. Their treatment explicitly extends to models that abstain. Quach et al. adapt the framework to generative language models with a stopping rule for sampling candidates and a rejection rule for removing low-quality ones, proving the returned set contains at least one acceptable answer with high probability while staying small on average [12].

What the evidence says

The table below compares the abstention mechanisms available to a typed-decision system, ordered roughly from cheapest to most demanding. Every row is independently deployable; the rows are complements, not rivals.

Three empirical findings cut across these rows. First, abstention does not come free with scale. AbstentionBench evaluates 20 frontier LLMs across 20 datasets and 6 abstention scenarios (unknown answers, underspecification, false premises, subjectivity, stale information, and related cases) and reports that abstention is unsolved and scaling is of little use [13]. Its README adds that the benchmark ships data on Hugging Face at facebook/AbstentionBench with human-validated judges for scalable evaluation [14].

Second, the kind of training matters more than the amount. AbstentionBench reports that reasoning fine-tuning degrades abstention by 24% on average, even in the math and science domains the reasoning models are trained for, and that a crafted system prompt boosts abstention in practice without resolving the underlying inability to reason about uncertainty [13]. R-Tuning points the other way: training explicitly for refusal improves both refusal behavior and calibration [9]. Abstention looks like a trainable skill with its own data requirements, not a side effect of general capability.

Third, the typed-decision evidence is behavioral and cautionary. The independent decision-model benchmark (DMB) by nibzard includes forced-uncertainty items and reports that every tested LLM admits ignorance on 97.3 to 100% of them, while Jev admits ignorance on 49.7% [15]. DMB also reports Jev’s expected calibration error (ECE) as 0.246 against an LLM range of 0.039 to 0.122. These are third-party measurements of a closed application programming interface (API), covered in context by the Jev survey in this graph. The narrow point for this paper: a typed-decision API without an explicit abstention path bluffs when forced, and its confidence numbers cannot gate what its interface never declines.

Analysis

Given the evidence, there are four decisions to make when adding abstention to a typed-decision system: the surface, the trigger, the threshold, and the score.

The surface is what “I do not know” looks like. The minimal honest surface has three properties: a machine-readable abstention signal distinct from any label, a reason code from a small closed set (unanswerable input, out-of-distribution state, below-threshold confidence, over-limit cardinality), and the counterfactual the caller needs (the answer the model would have given, with its confidence). SQuAD 2.0 shows the value of the first property: abstention is gradable because it is a distinct output, not a low score [5]. DMB shows the cost of its absence: when refusal is not in the schema, the model fills the schema with a bluff [15]. The alternative, keeping the schema fixed and letting callers gate on confidence, is Laya’s documented pattern, and it works only if confidences are honest into the far tail, which is where Laya’s own numbers fail (zero accuracy at 0.952 mean confidence on Khmer, per the Laya teardown in this graph). An explicit abstain output and caller-side thresholding are complements.

The trigger is which signal drives refusal. For closed choice heads the cheapest triggers are max softmax probability and normalized entropy, both computable from the output distribution with no extra parameters, and the Hendrycks baseline justifies max-softmax as the default to beat [3]. A learned selection head in the SelectiveNet style is the next step up: it can use signals the distribution alone does not carry, such as input-side OOD features, at the cost of training support and its own calibration [2]. A P(IK) style “I know this” logit alongside the decision is the typed-decision analogue of Kadavath’s approach, with the same caveat: cross-task calibration of the knowledge signal is the documented weak point [7]. Conformal prediction sits apart because it changes the output from a point to a set [11]; for a choice head, the conformal analogue of abstention is returning a large set or the full option list, which callers must agree in advance to read as “I do not know.”

The threshold is where the operating point gets set, on held-out data with a stated cost model. Selective classification makes this explicit: pick a target risk, then reject to meet it [1]. The target risk comes from the relative cost of a wrong answer versus an abstention, which is application-specific and cannot be read off a benchmark. Two practices follow. First, fit thresholds per bucket (question type, option count, language), the same grouping the Maestro Flash proposal uses for temperatures, because a single global threshold is simultaneously too lax on hard slices and too strict on easy ones. Second, report the full risk-coverage curve rather than a single operating point, so a change in application costs does not require rerunning the evaluation.

The score is how evaluators grade abstention without punishing honesty. Three metrics cover the space. Selective risk (error rate on answered items) and coverage (fraction answered) describe the tradeoff curve [1]. Abstention precision and recall, measured on items labeled unanswerable by construction, describe refusal quality directly: precision is the fraction of refusals that were truly unanswerable, recall the fraction of unanswerable items refused. SQuAD 2.0 is the template for that slice: unanswerability defined by the task setup, adversarially written to resemble answerable items, graded by exact output [5]. Finally, a cost-weighted score combines the three outcomes (correct, wrong, abstain) with application costs into one number for operating-point selection. AbstentionBench’s judge-based protocol (LLM judges validated against human annotation [14]) scales grading to open-ended refusals, but for typed decisions it should be unnecessary: a closed output schema makes abstention exactly checkable without a judge.

One more consideration is what not to count as abstention. A 400 error response at over-limit cardinality is input validation, not uncertainty: it refuses malformed requests, not uncertain ones. Conflating the two inflates refusal statistics with cases no uncertainty method could answer. Evals should separate request-level rejection (over-limit, malformed, empty state) from decision-level abstention (well-formed input, model declines) and score them independently.

Implications for small decision models

For Jev, the implication is an interface gap rather than a quality verdict. DMB’s third-party measurement finds Jev admits ignorance on 49.7% of forced-uncertainty items where LLMs admit 97.3 to 100%, with ECE 0.246 against the LLMs’ 0.039 to 0.122 [15]. If a future API revision adds an explicit abstain output with reason codes, the right benchmark is a risk-coverage curve over the existing suites plus a SQuAD 2.0 style unanswerable slice, scored with abstention precision and recall. Until then, callers gating Jev on confidence should treat the gate as unvalidated on forced-uncertainty inputs.

For Laya, the path is incremental. Laya already computes the raw material for abstention: per-question distributions, entropy-based confidence (1 minus normalized entropy), per-bucket fitted temperatures, and a documented confidence-gating pattern. Its README describes typed decisions over 100-plus languages in a single forward pass at 33 ms per question and 7.2 ms batched on an NVIDIA T4 graphics card [16], and its public checkpoints are convaiinnovations/laya (ModernBERT-large, 421M) and convaiinnovations/laya-multilingual (mmBERT-base, 322M), both Apache-2.0 with text-classification pipeline tags on Hugging Face [17]. What Laya lacks is a first-class abstain output: refusal today is caller-side thresholding, and the teardown in this graph documents confidence saturating in failure regions (including an action-head probability stuck at 1.0). Three additions suggest themselves in increasing order of ambition: publish per-bucket abstention thresholds fitted on held-out data alongside the per-bucket temperatures; add an explicit abstain decision with reason codes to the schema; and, if training is revisited, add a SelectiveNet-style selection head or refusal-aware tuning data so the model learns the refusal boundary instead of leaving it to a post-hoc cutoff.

For Velofy’s proposed Maestro Flash, abstention should be designed in from the start, since the proposal already contains every hook it needs. Three additions would make it first-class. First, add an abstain output to the schema for every head, with a closed reason-code set and a counterfactual field, so refusal is gradable without a judge. Second, add an unanswerable slice to the evaluation plan: underspecified states, contradictory options, and out-of-language inputs by construction, scored with abstention precision and recall alongside the risk-coverage curve. Third, treat the trigger as an experiment: compare max-softmax thresholding, entropy thresholding, a learned selection head, and refusal-aware tuning data on the same backbone, with the Hendrycks baseline as the floor to beat. This is measurement and interface design, which is what a proposal-stage project can contribute. Maestro Flash has no weights and no results, and nothing here claims otherwise.

Limitations

This survey ran no experiments; every empirical claim is attributed to its source. Most abstention evidence comes from two regimes that bracket but do not include the target: vision classifiers with hundreds of classes (selective classification) and generative LLMs with open-ended outputs (self-evaluation, refusal tuning, AbstentionBench). Small encoders with closed typed outputs are the gap in the middle, and no benchmark was found that measures their abstention directly. The DMB forced-uncertainty result is the closest behavioral data point, and it covers one closed API plus LLM baselines, not open encoder heads.

Judge-based scoring, as used by AbstentionBench with human-validated LLM judges, risks circularity: the judge’s own uncertainty handling bounds what it can grade. Exact-match grading on closed schemas avoids that problem, but the recommendation is untested here. Conformal guarantees assume exchangeability between calibration and deployment data, which fails under the distribution shifts that motivate abstention in the first place; the guarantee degrades exactly when it is most needed. Finally, every cost model here is application-specific, so no universal threshold or single-number metric can be recommended. Each deployment must fit its own operating point on its own held-out data.

Open questions

1. What is the cheapest abstention trigger that beats max-softmax thresholding on a small typed-decision encoder, and on which slices does it win?
2. How should a typed-decision benchmark construct unanswerable items (underspecified state, contradictory options, out-of-language input) so that abstention precision and recall are exactly gradable without a judge?
3. At what point in the training stack should refusal be learned (pretraining signal, refusal-aware tuning data, selection head, post-hoc threshold), and how do those choices interact with proper-scoring-rule calibration?

References

1. Geifman and El-Yaniv, Selective Classification for Deep Neural Networks. arXiv:1705.08500
2. Geifman and El-Yaniv, SelectiveNet: A Deep Neural Network with an Integrated Reject Option. arXiv:1901.09192
3. Hendrycks and Gimpel, A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. arXiv:1610.02136
4. Guo et al., On Calibration of Modern Neural Networks. arXiv:1706.04599
5. Rajpurkar et al., Know What You Don’t Know: Unanswerable Questions for SQuAD. arXiv:1806.03822
6. The Stanford Question Answering Dataset official site (SQuAD 2.0 explorer and evaluation). https://rajpurkar.github.io/SQuAD-explorer/
7. Kadavath et al., Language Models (Mostly) Know What They Know. arXiv:2207.05221
8. Lin et al., Teaching Models to Express Their Uncertainty in Words. arXiv:2205.14334
9. Zhang et al., R-Tuning: Instructing Large Language Models to Say “I Don’t Know”. arXiv:2311.09677
10. R-Tuning official repository (NAACL 2024 Outstanding Paper Award note and method summary). https://github.com/shizhediao/R-Tuning
11. Angelopoulos and Bates, A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. arXiv:2107.07511
12. Quach et al., Conformal Language Modeling. arXiv:2306.10193
13. Kirichenko et al., AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions. arXiv:2506.09038
14. AbstentionBench official repository and benchmark documentation. https://github.com/facebookresearch/AbstentionBench
15. nibzard/decision-model-benchmark, independent typed-decision benchmark (forced-uncertainty and calibration findings). https://github.com/nibzard/decision-model-benchmark
16. NandhaKishorM/laya, open typed-decision engine README (architecture, checkpoints, latency figures). https://github.com/NandhaKishorM/laya
17. convaiinnovations/laya model card on HuggingFace (license, tags, pipeline). https://huggingface.co/convaiinnovations/laya
18. scikit-learn user guide, Probability calibration (official documentation). https://scikit-learn.org/stable/modules/calibration.html

← All papers