Calibrating Sigmoid Label Scorers for Selective Automation

Status: SURVEY. This paper synthesizes published calibration literature, public READMEs, source files, and benchmark reports. It reports no new experiments. Every number below comes from a source fetched for this paper and is attributed inline. Velofy has not trained or shipped a decision model, and every Maestro Flash statement below is a proposal target.

The question

The parent paper in this graph, Zero-Shot Label Scoring in the GLiNER Family, ends with a calibration question: what is the cheapest recipe that gives a GLiNER-style per-label sigmoid scorer usable selective-automation curves, and does per-option-count temperature fitting suffice, or does the independent-versus-softmax structure of the scores require a proper-scoring-rule training change? This paper answers that question from published evidence only.

The terms need pinning down first. A GLiNER-style scorer matches each label representation against the input, passes each score through an independent sigmoid, and trains with binary cross-entropy, so each output reads as the probability that the label applies. A softmax scorer instead normalizes one logit vector into a distribution that sums to one. Selective automation means routing by confidence and reporting maximum threshold-realizable coverage at a fixed empirical error budget, the metric the independent Jev-versus-GLiNER pilot defines. Per-option-count temperature fitting means learning one temperature per label-count bucket on held-out data, the recipe the open Laya engine ships. A proper-scoring-rule training change means replacing or augmenting cross-entropy with a loss uniquely minimized by reporting true beliefs.

The short answer: temperature fitting per count bucket is the cheapest recipe that works, and it suffices for honest confidence scale, but not for usable selective-automation curves on its own. Two further post-hoc ingredients are needed: an explicit single-label decision rule over the independent sigmoids, and a ranking check, since no monotonic rescaling creates separation the scorer lacks. A training change is warranted only when the ranking check fails.

First principles

A confidence score has two separate jobs. It must rank: wrong answers should score lower than right ones. It must also be on the right scale: answers stated at 0.8 should be right about 80 percent of the time. A selective automation threshold only decides where to cut a sorted list, so coverage at a fixed error budget depends on ranking. Calibration metrics such as expected calibration error (ECE) measure scale.

A monotonic rescaling cannot change order. For one sigmoid output, dividing the logit by a positive temperature keeps every item in the same order. Platt and isotonic maps are monotonic too. So post-hoc calibration can fix scale, but it cannot separate right answers from wrong ones that the raw score already mixes up.

K sigmoids are not a decision distribution. With K labels, each sigmoid answers “does this label apply?” on its own, and nothing forces the K outputs to sum to one. A single-label decision needs a different number: the probability that the top label is correct. Getting that number takes an explicit rule, such as renormalizing the calibrated sigmoids or fitting a temperature on the argmax margin.

A softmax gets that rule for free. Normalizing one logit vector already produces a distribution over the options, so its top probability speaks about the decision. Dividing the vector by a temperature changes only its sharpness and never the argmax.

So the fix depends on which property is broken. A scale error is cheap: one parameter per bucket, fitted on held-out data. A ranking error needs a better scorer, through more label evidence or a training change. The ordering this paper recommends follows from that split.

Background

Calibration is the property that stated confidence matches empirical accuracy. Guo et al. (arXiv:1706.04599) define it as P(correct given confidence p) equals p for all p, measured with expected calibration error (ECE). Their finding is that modern networks are accurate but overconfident, and the miscalibration is intrinsically low dimensional, which is why a single scalar fix works.

Temperature scaling is that scalar fix. For logits z and temperature T, the calibrated distribution is softmax(z over T), with T learned by minimizing log-loss on held-out validation data. The scikit-learn calibration documentation describes exactly this procedure. Two properties matter: it is accuracy preserving (a positive scalar cannot change the argmax), and it is cheap (one parameter, one validation pass, no retraining). Guo et al. report that it outperforms all other compared methods on their vision tasks, including strictly more general vector and matrix variants, and that vector scaling recovers essentially the same constant solution, which is the evidence behind the low-dimensional claim.

Strictly proper scoring rules are the training-side alternative. Gneiting and Raftery define a scoring rule as proper when a forecaster maximizes expected score by issuing their true belief distribution, and strictly proper when that maximum is unique. Their review names the logarithmic, spherical, and quadratic (Brier) scores as canonical examples. Since cross-entropy is the log score, standard training already optimizes a proper rule; proposals to add Brier or spherical terms, or to train with reinforcement learning (RL) against a rule combination, try to make the objective care about the full reported distribution. Laya’s public source makes this concrete: its proper_reward returns the log score plus 0.5 times the spherical score, minus 1.0 times the ranked probability score (RPS) on ordinal questions.

Sigmoid label scorers are the architecture under test. The GLiNER paper (arXiv:2311.08526) defines its matching score as a sigmoid over the dot product of a span representation and a label embedding, trains with binary cross-entropy, and states the score reads as the probability the span has the type. Usage examples in the fetched GLiNER, GLiClass, and GLiNER2 READMEs then apply a fixed threshold, almost always 0.5. GLiClass (arXiv:2508.07662) adds a temperature tau inside its dot-product scorer, but no fetched source reports fitted tau values or any calibration curve for a GLiClass checkpoint, so tau reads as a learned sharpness control, not a calibration recipe. The asymmetric loss paper (arXiv:2009.14119) describes the same structure from the multi-label side: each logit independently activated by a sigmoid, with the paper’s contribution being management of the resulting positive-negative imbalance.

Selective classification is the evaluation frame. Geifman and El-Yaniv (arXiv:1705.08500) frame the goal as a classifier with a dial: the operator sets a risk level, and the classifier rejects instances as needed to grant it while keeping coverage high. Their headline result is 2 percent error in top-5 ImageNet classification guaranteed with probability 99.9 percent at almost 60 percent coverage. SelectiveNet (arXiv:1901.09192) trains the rejection mechanism jointly with the classifier, reporting improved risk-coverage tradeoffs over thresholding a pre-trained network. The Jev-versus-GLiNER pilot operationalizes this frame with threshold-realizable coverage: pick the confidence threshold subject to the error budget, report the fraction automated.

Per-option-count temperature fitting is the cheapest published recipe that adapts calibration to label-count scaling. Laya’s source defines temp_bucket with boundaries at 2, 3 to 5, 6 to 10, and 11 plus, fits one temperature per (question type, bucket) pair on held-out data, and clamps applied values to [0.5, 5.0]. The clamp exists because of a real incident the source comments on: a shipped choice:11+ value of 0.1006 would multiply logits by about 10x, so the client clamps instead of trusting the fit. Laya’s README reports that refitting moves mean ECE from 0.466 to 0.081 on its English checkpoint, and its BENCHMARKS.md calls the refit the single highest-value fix available. Those two numbers are the existence proof that cheap post-hoc fitting works on a small decision encoder. This paper asks what it leaves unfixed.

What the evidence says

The evidence divides into four groups: post-hoc methods work, post-hoc methods have limits, training changes help innate calibration, and the score structure creates a decision-rule problem.

First, post-hoc calibration works. Guo et al. report ECE on CIFAR-100 with ResNet 110 falling from 16.53 percent to 1.26 percent with temperature scaling, and on CIFAR-10 with ResNet 110 from 4.6 to 0.83 percent, all with accuracy unchanged. Laya’s README reports the decision-encoder analogue: mean ECE 0.466 before fitting, 0.081 after. Komisarenko et al. (arXiv:2408.11598) extend the point: their two-parameter focal temperature scaling beats plain temperature scaling on all three tested image datasets, reporting CIFAR-100 ECE of 2.13 against 3.01. A small held-out fit buys most of the available ECE reduction at negligible cost.

Second, post-hoc calibration has limits, and three are load-bearing here. Kull et al. (arXiv:1910.12656) show temperature scaling reduces overall ECE while worsening calibration in the middle range: on their balance-scale example it degrades the [0.2, 0.6] interval per class, and on classwise ECE (cw-ECE) it scores 0.15 against 0.12 for both Dirichlet calibration and one-versus-rest isotonic regression. That confidence-versus-classwise distinction matters directly for per-label sigmoids, where every output is a classwise claim. Guo et al. add that temperature scaling fails outright on Reuters, where only a binning method helps, so the scalar fix is not universal. And Zhang et al. (arXiv:2003.07329) warn that histogram-based ECE itself can mislead on little data; recipes tuned on small held-out slices should report uncertainty, not just point ECE.

Third, training changes improve innate calibration: the model needs less correction afterward. Mukhoti et al. (arXiv:2002.09437) show focal-loss training moves the optimal post-hoc temperature toward 1.0, their marker of an innately calibrated model. On CIFAR-100 with ResNet 50, cross-entropy starts at 17.52 ECE and needs T equal 2.1 to reach 3.42, while focal loss with gamma 3 starts at 5.13 and needs only T equal 1.1 to reach 1.97. Komisarenko et al. explain why: focal loss decomposes into a confidence-raising map plus a proper loss, pushing training predictions under-confident so the generalization gap corrects in the right direction. On the multi-label side, Ridnik et al. report asymmetric loss reaching 86.6 percent mean average precision (mAP) on MS-COCO, 2.8 points above the previous top, with focal at 85.1 against 84.0 for cross-entropy. These are ranking gains from a training change that post-hoc scaling cannot replicate. Whether Laya’s proper-scoring-rule RL beats cross-entropy plus temperature scaling on the same encoder is UNVERIFIED: no fetched source reports that ablation.

Fourth, the independent-versus-softmax structure is a decision-rule problem first and a calibration problem second. The scikit-learn documentation states the mechanics plainly: isotonic and sigmoid calibrators support only one-dimensional output, so multiclass calibration fits one calibrator per class one-versus-rest (OvR), the probabilities do not necessarily sum to one, and a postprocessing step normalizes them. Temperature scaling instead works on the logit vector with softmax, so it naturally yields a distribution. A GLiNER-style scorer sits on the one-versus-rest side: K independent sigmoids, no sum constraint, decisions by fixed threshold. The pilot shows what that costs. On AG News (4 labels) it reports Jev accuracy 0.910 against GLiNER2.5 at 0.700, with coverage at 5 percent error of 0.830 against 0.240; on Banking77/BTZSC (72 labels), 0.870 against 0.610, with coverage 0.860 against 0.270. Yet on DAIR Emotion (6 labels), where accuracy ties (0.480 against 0.440), GLiNER2.5 has better probability scores: Brier 0.668 against 0.846 and negative log-likelihood (NLL) 1.381 against 5.588, with Jev assigning zero probability to truth on 16 percent of examples. Better average probability scores with threefold worse selective coverage means the binding constraint is ranking and separation, not confidence scale. No monotonic rescaling manufactures separation the scorer lacks.

Two rows deserve a note. The conformal row (arXiv:2004.10181) offers guaranteed coverage on prediction sets, including tree-structured sets for multilabel problems, but sets are not single decisions with confidences, so it answers a neighboring question. SelectiveNet is the only method whose training objective is the selective curve itself, and its full training loop with a rejection head makes it the expensive end of the spectrum, not the cheap recipe this node seeks.

Analysis

The cheapest recipe the evidence supports has four steps, all post-hoc. First, fit one temperature per (question type, option-count bucket) on held-out data, using Laya’s boundaries (2, 3 to 5, 6 to 10, 11 plus) as a starting grid, and clamp applied values to [0.5, 5.0]. Second, write down the single-label decision rule explicitly: argmax over the calibrated sigmoids with a stated normalization, since K independent probabilities are not a decision distribution. Third, measure threshold-realizable coverage at a fixed error budget alongside Brier, NLL, and ECE, since the pilot shows Brier and NLL can favor one system while coverage favors the other threefold. Fourth, report per-bucket curves, because Laya’s shipped 0.1006 in the 11-plus bucket proves the logit distribution shifts with option count.

This recipe suffices when miscalibration is scale-only, the common case in the literature. Guo et al. show one scalar removes most ECE across vision and natural language processing tasks; Laya shows the same for a decision encoder with count buckets; Komisarenko et al. show a two-parameter extension squeezes further. Total cost is one small labeled held-out set and a grid search over T. Nothing fetched suggests a GLiNER-style checkpoint needs more to produce honest per-label probabilities, provided its ranking is adequate.

The recipe fails where structure, not scale, is the problem, and the evidence names four such cases. First, poor ranking: GLiNER2.5’s 0.240 and 0.270 coverage against Jev’s 0.830 and 0.860 cannot be rescaled away. The fix is a better scorer (GLiClass reports 8-shot label examples lifting Banking77 edge from 0.0255 to 0.6644; or a loss change), not a better temperature. Second, the classwise-versus-confidence gap: temperature fitted to top-label NLL need not calibrate mid-range per-label probabilities, and Kull et al. show it can worsen that range. The fix is per-class Platt or isotonic maps when data allows (about 1000 samples per sklearn guidance). Third, count-dependent shift: GLiClass trains tau inside the scorer, and no fetched source reports whether its sharpness drifts with label count, so deployments must check the 11-plus bucket rather than assume it. Fourth, threshold arbitrariness: the 0.5 default in every fetched README is convention, not a calibrated boundary under class imbalance.

The verdict on the node’s second clause is therefore: the independent-versus-softmax structure requires a decision-rule change, not a training change. Calibrating each sigmoid gives honest P(label applies) per label; single-label automation needs honest P(argmax is correct), a different quantity. The cheapest bridge is post-hoc: renormalize the calibrated sigmoids into a distribution, or fit temperature against the argmax margin on held-out data, then gate on the result. Softmax-native scorers like Laya’s get this bridge for free, since their temperature already operates on the decision distribution. This also explains why Laya’s entropy confidence, 1 minus H(p) over log(k), is a coherent gate for Laya but has no direct analogue for unnormalized sigmoids: entropy of K independent probabilities is not the uncertainty of one decision.

A training change is warranted only after the post-hoc ceiling is measured and found wanting. The supported ordering is: temperature per bucket first (minutes, no retraining); per-class or Dirichlet maps second (needs more held-out data); decision-rule redesign third (needs a stated error budget); and only then focal, asymmetric, or proper-rule training (needs a training loop). Mukhoti et al. show training changes move the optimal temperature toward 1.0 rather than removing post-hoc fitting, so even the training path ends with a temperature fit. Skipping straight to proper-scoring-rule RL risks paying for a training loop to fix a scale problem a scalar would have solved.

Implications for small decision models

For Jev, the implications are behavioral only, since its architecture and training are undisclosed. The independent decision-model-benchmark (DMB) reports Jev at ECE 0.246 against 0.039 to 0.122 for tested large language models (LLMs), with 13 percent of choices flipping under permutation and a sharp 255-choice cap. The pilot reports strong selective coverage where accuracy is strong (0.830 and 0.860 at 5 percent error) but poor DAIR Emotion probabilities (Brier 0.846, NLL 5.588, zero mass on truth for 16 percent). Jev’s curves are good when ranking carries them and bad when probabilities must, which is exactly where the cheap recipe helps most. But Jev is a closed application programming interface (API) with an UNVERIFIED confidence pipeline, so there is no public mechanism to apply it to. The practical implication, documented in Jev and the Typed-Decision Landscape, is to treat Jev as a behavioral reference and fit gating thresholds on your own data.

For Laya, this paper’s recipe largely describes what Laya already does, which makes Laya the validation case and its residuals the warning. Validated: per-bucket temperatures on held-out data, clamped to [0.5, 5.0], taking mean ECE from 0.466 to 0.081. Residuals, all from fetched Laya sources: both checkpoints ship over-confident, the multilingual checkpoint ships with no fitted temperatures, the 11-plus fit produced a sharpening value the clamp had to catch, and entropy confidence differs from the argmax probability a gate most naturally consumes. The sibling teardown, Laya: An Open Typed-Decision Engine, Examined, adds a live-run warning: the undocumented action head saturated at 1.0 across all test answers, so gating should use the documented entropy confidence until that issue is resolved. Laya’s trajectory confirms the ordering: proper-rule RL training exists, but its biggest reported calibration win came from the post-hoc refit.

For Velofy’s proposed Maestro Flash, which has no weights and no results, the evidence converts into design constraints. Ship per-bucket temperature fitting with clamping from the first checkpoint. Define the single-label decision rule over sigmoid outputs in the interface specification before any benchmark, since the sklearn one-versus-rest mechanics show normalization is not optional. Measure threshold-realizable coverage as a first-class metric, since the pilot shows it can disagree with accuracy and probability scores threefold. Treat few-shot label examples as a ranking input on Banking77-scale tasks before blaming the encoder, since GLiClass reports 8-shot recovery from 0.0255 to 0.6644. And settle backbone choice by matched experiment, since Minderer et al. (arXiv:2106.07998) find architecture is a major determinant of calibration properties. None of these is a Maestro Flash result. Each is a test the proposal must pass, alongside the training-side proposal in Calibrated Decisions via Proper-Scoring-Rule RL.

Limitations

No experiment was run: all numbers are cited from fetched papers, READMEs, source files, model APIs, and benchmark reports, none reproduced here. The pilot rests on 300 examples with 100 per condition, which supports its bootstrap intervals but not fine-grained claims at 72 labels, and the DAIR coverage figures (0.000 against 0.020) show how noisy threshold-realizable coverage gets near chance accuracy. ECE figures use different binnings and splits, so cross-paper ECE comparisons are approximate. No fetched source fits a post-hoc temperature on any GLiNER-family sigmoid checkpoint, so the central prescription is supported by analogy from Laya and the general literature, not by direct measurement; GLiClass’s in-scorer tau values are UNVERIFIED. Jev internals are UNVERIFIED throughout. Fetched Hugging Face counts (GLiNER2-base 445,822 downloads, top GLiClass checkpoint 14,938) measure reuse, not quality. Finally, this paper covers the single-label case in detail; multilabel selective automation with set-valued outputs is a separate problem where conformal methods, not temperature scaling, are the natural tool.

Open questions

1. How much selective-automation coverage does per-bucket temperature fitting buy on a GLiNER-style sigmoid checkpoint: fitting one temperature per option-count bucket and reporting threshold-realizable coverage at 5 percent error before and after, does the pilot’s threefold coverage gap close or persist?
2. Which single-label decision rule over independent sigmoids gives the best risk-coverage tradeoff: argmax over raw sigmoids, argmax over renormalized sigmoids, or a temperature fitted directly on the argmax margin, and does the winner vary with option count?
3. When the post-hoc ceiling is reached, which training change improves selective coverage per unit cost on a frozen small encoder: focal or asymmetric loss substitution, proper-scoring-rule policy gradients, or few-shot example-augmented label representations without any weight updates?

References

1. Guo et al., On Calibration of Modern Neural Networks, https://arxiv.org/abs/1706.04599
2. Kull et al., Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration, https://arxiv.org/abs/1910.12656
3. Mukhoti et al., Calibrating Deep Neural Networks using Focal Loss, https://arxiv.org/abs/2002.09437
4. Komisarenko et al., Improving Calibration by Relating Focal Loss, Temperature Scaling, and Properness, https://arxiv.org/abs/2408.11598
5. Geifman and El-Yaniv, Selective Classification for Deep Neural Networks, https://arxiv.org/abs/1705.08500
6. Geifman and El-Yaniv, SelectiveNet: A Deep Neural Network with an Integrated Reject Option, https://arxiv.org/abs/1901.09192
7. Zaratiana et al., GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer, https://arxiv.org/abs/2311.08526
8. Stepanov et al., GLiClass: Generalist Lightweight Model for Sequence Classification Tasks, https://arxiv.org/abs/2508.07662
9. Ridnik et al., Asymmetric Loss For Multi-Label Classification, https://arxiv.org/abs/2009.14119
10. Minderer et al., Revisiting the Calibration of Modern Neural Networks, https://arxiv.org/abs/2106.07998
11. Zhang et al., Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning, https://arxiv.org/abs/2003.07329
12. Angelopoulos et al., Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction, https://arxiv.org/abs/2004.10181
13. Gneiting and Raftery, Strictly Proper Scoring Rules, Prediction, and Estimation (author-hosted PDF), https://sites.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf
14. scikit-learn documentation, Probability calibration, https://scikit-learn.org/stable/modules/calibration.html
15. GLiNER repository README (urchade/GLiNER), https://github.com/urchade/GLiNER
16. GLiClass repository README (Knowledgator/GLiClass), https://github.com/Knowledgator/GLiClass
17. GLiNER2 repository README (fastino-ai/gliner2), https://github.com/fastino-ai/gliner2
18. Laya repository README and BENCHMARKS.md (NandhaKishorM/laya), https://github.com/NandhaKishorM/laya
19. Laya calibration source, laya/common.py temp_bucket and proper_reward, https://github.com/NandhaKishorM/laya/blob/main/laya/common.py
20. AbdelStark jev-benchmarks pilot comparing Jev with GLiNER2.5, https://github.com/AbdelStark/jev-benchmarks
21. nibzard decision-model-benchmark (DMB), https://github.com/nibzard/decision-model-benchmark
22. Hugging Face model API, GLiNER family search, https://huggingface.co/models?search=gliner

← All papers