The Smallest Selection Head That Beats Thresholding

This paper answers one question from Where Refusal Should Be Learned: what is the smallest selection-head or abstain-class design that beats per-bucket post-hoc thresholding on risk-coverage for a small typed-decision encoder, and does the winning design change the optimal temperature grouping? It does not repeat the parent paper’s placement taxonomy or its calibration analysis. All arXiv identifiers (IDs) below were verified against their arXiv abstract pages, and every number is attributed inline to a source fetched for this paper. Velofy has trained no model for this paper and ships none with it.

The question

The parent paper recommends a stack order (proper-scoring-rule base, refusal-aware tuning data, per-bucket post-hoc gate, learned head only if the post-hoc ceiling binds) and leaves the conditional open. This paper takes it up: suppose we add a head. What is the smallest thing we can add that actually wins?

Three terms need pinning down. A small typed-decision encoder is a ModernBERT-class bidirectional model that answers predeclared choice, score, or boolean questions in a single forward pass, the regime described in the Laya teardown and the Jev survey. Risk-coverage is the selective-classification tradeoff: the error rate on answered items (risk) plotted against the fraction of items answered (coverage). A head wins only if it lowers risk at matched coverage, or raises coverage at matched risk, on the same backbone and the same evaluation slices. Per-bucket post-hoc thresholding is the floor to beat: one fitted temperature plus one fitted cutoff per question-type and option-count group, both fit on held-out data with no weight updates. Laya’s README documents the concrete form this takes in practice: per-type and bucket-specific temperatures with a [0.5, 5.0] clamp at load [16].

Smallest is two-dimensional: fewest added parameters, and least disruption to training and inference. An abstain class that adds one output row and one loss term is smaller than a selection head with its own hidden layer even before counting parameters, because it changes no serving path except the score the gate reads. The minimal winner differs by regime, so in-domain and shift-heavy settings are answered separately below.

First principles

A selective classifier has two parts: a predictor that picks an answer, and a gate that decides whether to release it. The gate is a score plus a cutoff. Every design in this paper differs only in where that score comes from and when it is trained.

A post-hoc gate can only use information already in the model’s outputs. Max-softmax, softmax response, and per-bucket temperature scaling are all functions of the returned logits and the bucket identity. If a wrong answer produces logits that look like those of a right answer, no rescaling or cutoff can separate the two.

So a learned head can win in only two ways. It can read information the outputs discard, such as internal features or input lengths. Or it can change training so that the outputs themselves carry more of the error signal. Detached heads use only the first route. Abstain classes use only the second. Joint selection heads and confidence branches use both.

The second route has a side effect. A temperature is fit to one particular output distribution. A design that changes that distribution makes the old temperatures stale; a design that leaves it alone keeps them valid.

Winning is judged on the risk-coverage curve at matched coverage, not at one favorable operating point. The application’s cost of a wrong answer, set against the cost of an abstention, decides which coverage matters. A fair comparison therefore reports the whole curve.

Background

The reject option is old: the survey by Hendrickx et al. notes the field was already studied in 1970, and it formalizes the central split between ambiguity rejection (genuinely hard inputs) and novelty rejection (inputs from outside training) alongside evaluation strategies and learning architectures [12]. That split organizes this paper. Thresholding a calibrated confidence is the classical answer to ambiguity. Novelty is the case a scalar output confidence cannot see, because the failure lives in the input representation rather than in a close posterior.

The post-hoc floor has three layers. The oldest is the max-softmax baseline of Hendrycks and Gimpel: correctly classified examples tend to have greater maximum softmax probabilities than misclassified and out-of-distribution examples, and the authors show this simple signal is effective across vision, language, and speech tasks while noting it can sometimes be surpassed [5]. Geifman and El-Yaniv add guarantees: their softmax-response method (SR) rejects at test time to meet a desired risk level, reporting 2% error on ImageNet top-5 with probability 99.9% at almost 60% coverage [4]. Underneath both sits calibration: Guo et al. show modern networks are poorly calibrated and report that temperature scaling, a single-parameter variant of Platt scaling, is surprisingly effective on most datasets [6]. The toolkit form is documented by scikit-learn, where a sigmoid (Platt) or isotonic calibrator is fit on held-out outputs [7]. Per-bucket thresholding repeats this per slice: one temperature and one cutoff per group.

Learned heads come in two families. The abstain-class family folds refusal into the output distribution itself, turning an m-class problem into m+1 classes where the extra class means abstain, trained with a loss that prices answering against abstaining. The selection-head family keeps a separate scalar: a selection function trained jointly with the predictor (SelectiveNet), a confidence branch trained with a hint mechanism (DeVries and Taylor), an auxiliary network trained post-hoc on frozen features (ConfidNet), or a small detached calibrator trained on held-out behavior (Kamath et al.). The families differ in what they read (output probabilities only, versus internal features plus behavior), when they train (jointly, versus after the predictor is fixed), and whether they change the decision distribution the temperatures were fit on. That last difference is why the temperature-grouping half of the question matters, and it is addressed in the analysis.

What the evidence says

Every head claim is relative to the floor, and the floor is not naive. Max-softmax thresholding costs zero parameters and works across modalities, but it reads only the top probability and inherits base miscalibration [5]. SR adds a risk guarantee but remains a function of the network’s own softmax output [4]. Temperature scaling adds one scalar per group and is surprisingly effective on most datasets [6], yet Ovadia et al. bound the approach: under dataset shift, traditional post-hoc calibration falls short, while some model-marginalizing methods give surprisingly strong results [14]. Ensembles are one such method (calibration as good as or better than approximate Bayesian networks, higher uncertainty on out-of-distribution (OOD) inputs, demonstrated at ImageNet scale, at the cost of serving several models) [13]. And focal loss learns models that are already very well calibrated, reaching state of the art calibration with temperature scaling while preserving accuracy [15]. A head must beat a calibrated, possibly per-bucket threshold, not a strawman.

The abstain-class family has one canonical minimal design. Deep Gamblers transforms the m-class problem into m+1 classes, where the extra class means abstain, and trains with a loss based on the doubling rate of gambling; the authors report that it requires almost no modification to inference or architecture and achieves strong results on SVHN and CIFAR10 at various coverages [2]. The official code states the loss as l(x,y) = -log(o f(x)y + f(x){m+1}), where o is the payoff for a correct prediction and f(x)_{m+1} is the reservation (abstain) output used as the disconfidence score for gating [3]. The paper proves the problem is meaningful for o strictly between 1 and m, with higher o encouraging confident prediction. On SVHN at 0.90 coverage the authors report error 0.76 plus or minus 0.05 against 0.89 for SR and 0.82 plus or minus 0.01 for SelectiveNet; on CIFAR10, 2.19 plus or minus 0.12 against 2.89 for SR and 2.43 for SelectiveNet [2]. Two caveats: the operating point enters training through o (the code still sets the gate post-hoc via a coverage evaluation flag) [3], and low payoffs can collapse learning, which the code answers with cross-entropy pretraining by default (100 epochs on CIFAR10 when o is below 6.1) [3].

The joint confidence branch is only slightly larger. DeVries and Taylor add a branch after the penultimate layer; the paper states it contains one or more fully connected layers ending in a single sigmoid scalar c, so the minimal instance is one extra layer plus one output [8]. Training interpolates predictions toward the target by the confidence, p’ = c p + (1 - c) y, under a budget hyperparameter beta on the confidence penalty with an adaptive weight lambda [8]. The authors report this surpasses output-distribution-based confidence on OOD detection without additional labels or OOD examples [8]. Minimal in parameters, but not in machinery: it changes the loss and couples confidence to predictor training.

The detached calibrator holds the smallest verified win against thresholding. Kamath et al. study selective question answering (QA) under domain shift: a SQuAD-trained model faces mixtures of in-domain and OOD questions, where MaxProb abstention is overconfident and abstains too little on OOD inputs [11]. Their fix trains a calibrator to predict whether the QA model was correct, on held-out source data plus known OOD data from a different domain than the test OOD data. The calibrator could be any binary classifier; the one used is a random forest over seven features (passage length, predicted answer length, top five softmax probabilities) [11]. It answers 56.1% of questions at 80% accuracy, against 48% for direct probability thresholding [11]. No model weight is touched; the head is a small model over behavior plus two length features.

The auxiliary network trades size for a better learning target. ConfidNet attaches to the penultimate layer of a frozen classifier and learns the true class probability (TCP) rather than the maximum class probability (MCP), with guarantees for TCP in failure prediction; the authors report it consistently outperforms methods from MCP to Bayesian uncertainty [9]. The architecture used is five dense layers, though variants were tested with similar performance, so the five-layer choice is not a minimality claim and the floor of that architecture is UNVERIFIED here [9]. The official repository publishes the implementation [10]. The lesson is the target more than the size: learning TCP from features beats reading MCP from outputs.

The joint selection head is the largest and the strongest on pure risk-coverage. SelectiveNet trains prediction, selection, and an auxiliary head end to end over the covered domain, reporting consistently improved risk-coverage tradeoffs and new state of the art results [1]. The selection head used is a fully connected hidden layer with 512 units, batch normalization, and a rectified linear unit (ReLU), then a single sigmoid output; loss weights are alpha 0.5 and lambda 32; inference answers when selection is at least 0.5; the auxiliary head exists only for training [1]. The cost is rigidity: target coverage is a training constraint, so moving the operating point substantially means retraining.

Analysis

Five findings answer the question. The headline is regime dependence: the smallest winner under shift is identified and tiny, while the smallest in-domain winner against a properly calibrated per-bucket floor is still UNVERIFIED, with a clear prime candidate and a concrete experiment to settle it.

First, the exact comparison this question asks for is missing from the surveyed literature. Deep Gamblers beats SR and SelectiveNet at several coverages, but the SR baseline there is a single threshold over raw softmax outputs, not a per-bucket temperature-scaled threshold [1][2]. Since temperature scaling alone is surprisingly effective [6] and focal-loss training plus temperature scaling reaches state of the art calibration [15], an uncalibrated SR baseline is weaker than the floor a careful deployment fields. No surveyed paper puts a learned head and a per-bucket calibrated threshold on the same backbone. Any in-domain head-beats-floor claim is therefore UNVERIFIED, and the Maestro Flash paragraph below specifies the missing experiment.

Second, under shift or OOD-heavy mixtures, the smallest verified winner is the Kamath-style detached error predictor: seven features, no encoder retraining, 56.1% coverage against 48% for MaxProb at matched 80% accuracy [11]. The mechanism is visible. A scalar threshold reads one number the model produced itself, and models are overconfident on OOD inputs, so the threshold abstains too little where errors concentrate. The calibrator reads behavior the scalar discards (top-5 shape, lengths) and trains on observed errors from a different OOD distribution than the test one. DeVries and Taylor corroborate from penultimate features without OOD labels [8], and ConfidNet corroborates with a learned TCP target beating MCP through Bayesian methods [9]. Heads win where they read what the threshold cannot see.

Third, the minimal joint design is the Deep Gamblers abstain class: one extra output row (d weights plus one bias), one loss term, one payoff hyperparameter, no inference change except reading the reservation score [2][3]. It is smaller than every selection head in the table and beats both SR and SelectiveNet at several reported coverages. Its three costs are all about the operating point: o fixes risk appetite at training (deployments with different costs retrain or threshold the reservation post-hoc, which the code supports) [3]; low o risks a collapse where learning never starts, answered by cross-entropy pretraining [3]; and abstain mass inside the shared softmax changes the answered-conditional distribution, which leads to the temperature finding next.

Fourth, the winning design changes the optimal temperature grouping, but only in the joint family. For m+1, mass moves into the reservation output, so temperatures fit on the original m-way outputs no longer describe the gated distribution; the refit must happen on the answered-conditional distribution and per bucket, since abstain rates vary across buckets the way accuracy does. The resulting grouping may differ from the pre-abstain grouping, which makes it an empirical sub-result of the missing experiment. Detached heads have the opposite property: they leave the decision distribution untouched, so existing per-bucket temperatures stand and the head adds its own threshold on its own score [9][11]. SelectiveNet sits between: its prediction head was optimized over the covered domain, so any temperature must be fit at the deployed coverage [1]. Joint training couples the temperature question; detached heads preserve the separation the parent paper recommends keeping honest.

Fifth, the capacity ladder has a stopping rule. Start detached (a small classifier over top-k probabilities, pooled features, and bucket identity, trained on held-out in-domain plus known-OOD items): it is the only design with a verified win, it preserves temperatures, and it retrains fast when costs change. Move to joint m+1 only if the detached head fails in-domain on the same backbone, since m+1 is the cheapest joint change with head-to-head wins over post-hoc and joint baselines [2]. Move to SelectiveNet scale only with curve-level evidence that m+1 binds, since it is the most expensive and least re-pointable option [1]. ConfidNet’s similar-performing variants [9] hint at slack above the minimum, but the exact floor of a feature-reading head on a small encoder is UNVERIFIED.

Implications for small decision models

For Jev, the application programming interface (API) boundary constrains the answer. Jev is a closed API (see the Jev survey), so callers cannot train an abstain class or a joint head; only caller-side detached gates over returned confidences plus request-side features (option count, input length) are available. The independent decision-model benchmark (DMB) reports Jev at 76.3% on 77-way banking intent, median (p50) latency 264 to 276 ms flat from 2 to 255 options, a sharp 255-choice cap, expected calibration error (ECE) 0.246 against a large language model (LLM) range of 0.039 to 0.122, ignorance admitted on 49.7% of forced-uncertainty items where every tested LLM admits 97.3 to 100%, and a 13% permutation flip rate [18]. The raw material for a Kamath-style gate exists, but the ECE and forced-uncertainty figures say it is unvalidated where refusal matters. A caller-side error predictor trained on logged traffic with known-OOD items is the direct translation of the smallest verified winner; anything joint waits on a future API revision, UNVERIFIED today.

For Laya, each step reuses open artifacts. Laya is Apache-2.0 with a public checkpoint card (convaiinnovations/laya, tagged text-classification) [17], serves a single forward pass at 33 ms (7.2 ms per question batched on a T4), and trains with reinforcement learning (RL) against strictly proper scoring rules [16]. Notably, Laya already ships a learned selection-adjacent head that currently loses: the teardown documents an action head over pooled features, and the README reports act_probability reading 1.0 for almost every input, with logits running against correctness: an area under the receiver operating characteristic curve (AUROC) of 0.30 on 396 labeled decisions. It advises callers to gate on confidence instead (AUROC 0.77) [16]. The cheapest experiment that could flip that answer is the Kamath rung: a detached error predictor over top-k probabilities, pooled marker features, and bucket identity, trained on a held-out slice plus the unanswerable items the construction paper specifies, scored on per-bucket risk-coverage curves. If it cannot beat the gate in-domain, the next rung is an m+1 abstain output over the option markers with a per-bucket payoff sweep and an answered-conditional temperature refit.

For Velofy’s proposed Maestro Flash, which has no weights and no results, the contribution is a measurement plan that makes the missing comparison exist. First, fix the schema early: explicit in-schema abstain with reason codes (admits detached and joint heads) versus caller-side thresholding (post-hoc only). Second, hold one backbone fixed across four configurations (per-bucket temperature plus threshold; plus a detached Kamath-style head; plus m+1; all combined) on one unanswerable slice that is exactly gradable without a judge. Third, report full risk-coverage curves per bucket plus selective ECE at fixed coverage, never a single operating point, so cost changes re-read the same curves. Fourth, record the temperature protocol per configuration: answered-conditional refit for m+1, unchanged refit for the detached head, coverage-conditional refit for any joint selection head. If post-hoc matches the learned configurations, they are complexity without benefit; if one wins, the curves show where, at what coverage cost, and under which grouping.

Limitations

This survey ran no experiments; every empirical claim is attributed to its source. The synthesis spans vision classifiers, QA models, and surveys with different setups, not a controlled ablation, and small typed-decision encoders remain the gap in the middle. Numbers from different benchmarks do not share units: a coverage gap on QA mixtures and an error rate on SVHN neither compare nor transfer numerically to a typed-decision encoder. The central missing cell stands: no surveyed paper compares a learned head against per-bucket temperature-scaled thresholding on one backbone, so the in-domain minimal winner is UNVERIFIED. Every operating point depends on application costs, so no universal head size, payoff, or threshold is recommended.

Open questions

1. Does a Kamath-style detached error predictor (top-k probabilities plus pooled features plus bucket identity, trained on held-out in-domain plus known-OOD items) beat per-bucket temperature-plus-threshold on risk-coverage for a small typed-decision encoder, and how small can its feature set get before the win disappears?
2. Does a Deep Gamblers style m-plus-1 abstain class beat the post-hoc floor and the detached head at matched coverage on the same backbone, and does refitting temperatures on the answered-conditional distribution restore or change the optimal per-bucket grouping?
3. How much known-OOD or unanswerable training data does the winning head need to add novelty rejection without hurting ambiguity rejection, and does that transfer across held-out OOD families the way Kamath et al. report for QA?

References

1. Geifman and El-Yaniv, SelectiveNet: A Deep Neural Network with an Integrated Reject Option. arXiv:1901.09192
2. Liu et al., Deep Gamblers: Learning to Abstain with Portfolio Theory. arXiv:1907.00208
3. Deep Gamblers official code repository (loss statement, payoff and pretraining defaults, coverage evaluation). https://github.com/Z-T-WANG/NIPS2019DeepGamblers
4. Geifman and El-Yaniv, Selective Classification for Deep Neural Networks. arXiv:1705.08500
5. Hendrycks and Gimpel, A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. arXiv:1610.02136
6. Guo et al., On Calibration of Modern Neural Networks. arXiv:1706.04599
7. scikit-learn user guide, Probability calibration (official documentation). https://scikit-learn.org/stable/modules/calibration.html
8. DeVries and Taylor, Learning Confidence for Out-of-Distribution Detection in Neural Networks. arXiv:1802.04865
9. Corbiere et al., Addressing Failure Prediction by Learning Model Confidence. arXiv:1910.04851
10. ConfidNet official code repository. https://github.com/valeoai/ConfidNet
11. Kamath et al., Selective Question Answering under Domain Shift. arXiv:2006.09462
12. Hendrickx et al., Machine Learning with a Reject Option: A survey. arXiv:2107.11277
13. Lakshminarayanan et al., Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. arXiv:1612.01474
14. Ovadia et al., Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. arXiv:1906.02530
15. Mukhoti et al., Calibrating Deep Neural Networks using Focal Loss. arXiv:2002.09437
16. NandhaKishorM/laya README (latency figures, RLCD training, temperature clamp, act_probability behavior). https://github.com/NandhaKishorM/laya
17. convaiinnovations/laya model card on HuggingFace (license, pipeline tag). https://huggingface.co/convaiinnovations/laya
18. nibzard/decision-model-benchmark, independent typed-decision benchmark README. https://github.com/nibzard/decision-model-benchmark

← All papers