Grouping Calibration Parameters by Option Count

The question

This report answers one open question from the option cardinality survey: how should temperature or calibration parameters be grouped by option count so that expected calibration error (ECE) and flip rate stay flat as label sets grow. The parent survey names calibration drift with option count as one of its soft degradation mechanisms. It also notes that Laya BENCHMARKS.md describes refitting one temperature per question type and option count as its highest value fix, which admits one global temperature does not transfer across label set sizes. This report works out what the calibration literature and the available decision model measurements say about how fine that grouping should be, how it should be fitted and validated, and which half of the target it can actually move.

ECE measures the gap between predicted confidence and observed accuracy, usually by binning predictions and averaging the gap per bin. Flip rate measures how often the top answer changes when the same options are presented in a different order. ECE governs whether a confidence threshold means what it says, and flip rate governs whether the answer depends on presentation order. A central result of this report is that these two targets need different levers. Per count temperature grouping addresses ECE. It cannot move flip rate on fixed logits, because temperature scaling preserves the argmax within each ordering, so the flip half needs order robust training or inference time debiasing instead.

This report runs no new experiments. Velofy has not trained or shipped a decision model, and every Maestro Flash statement below is a proposal target. Every number below is attributed inline to a source fetched for this report.

Background

Guo et al., arXiv:1706.04599 describe temperature scaling as a single parameter variant of Platt scaling that learns one multiplicative factor for the inputs to the last softmax layer, and Guo et al. report that it is surprisingly effective at calibrating predictions on most datasets they test. Logits are divided by a fitted temperature T before the softmax: T above one softens, T below one sharpens. Ranking is unchanged, so accuracy is unchanged and only the probabilities move. That property is both the appeal and the limit of the method.

Nixon et al., arXiv:1904.01685 study choices in calibration measurement including confidence versus all probabilities, class conditionality, bin count, adaptive bins, and the norm used to compare accuracy to confidence. Across MNIST, Fashion MNIST, CIFAR-10/100, and ImageNet, Nixon et al. report that conclusions on the rank ordering of recalibration methods are drastically impacted by the choice of calibration measure. They report that conditioning on the class leads to more effective evaluations, and that the L2 norm improves optimization and rank correlation relative to L1. Grouping comparisons are therefore conditional on the ECE variant, which must be fixed first.

The scikit-learn calibration documentation describes two post hoc options: a sigmoid model following Platt and isotonic regression, which fits a nonparametric stepwise mapping. Kumar et al., arXiv:1909.10155 analyze the sample cost of this tradeoff: histogram binning needs on the order of B over epsilon squared samples against order one over epsilon squared for scaling methods, where B counts distinct output probabilities. Their scaling binning calibrator fits a parametric function first and then bins its values, which Kumar et al. report needs only order one over epsilon squared plus B samples, with 35 percent lower calibration error than histogram binning on CIFAR-10 and ImageNet.

Kull et al., arXiv:1910.12656 propose Dirichlet calibration for multiclass problems, which Kull et al. describe as equivalent to log transforming the uncalibrated probabilities, followed by one linear layer and softmax. Unlike temperature scaling, the linear map can change class rankings, not just confidence. Kull et al. report improved confidence ECE, classwise ECE, log loss, and Brier score across a wide range of datasets and classifiers.

Option count enters calibration through the softmax denominator. For fixed logits, each added option adds a term to the normalizer, so top probabilities fall mechanically as the label set grows even when ranking is perfect. Entropy based confidence has the same dependence: the Laya teardown reports Laya defines confidence as one minus entropy over log of k, normalizing by option count. A temperature fitted mostly on small sets will therefore miscalibrate large sets unless the parameter is grouped by count.

Flip rate has a separate literature. Zheng et al., arXiv:2309.03882 report selection bias on multiple choice questions across 20 models and three benchmarks, where models prefer specific option identifiers, and Zheng et al. attribute this bias primarily to token bias toward specific identifier tokens. Their PriDe method estimates that prior by permuting option contents on a small number of test samples and then debiases the rest. The permutation sensitivity report and the standard flip rate protocol in this graph cover flip counting.

Two further results bound expectations. Minderer et al., arXiv:2106.07998 revisit calibration on recent image classifiers and report that the most recent models, notably those not using convolutions, are among the best calibrated, with architecture a major determinant of calibration properties. Grouping is therefore a correction on top of a backbone specific baseline. Ovadia et al., arXiv:1906.02530 benchmark uncertainty under dataset shift and report that traditional post hoc calibration falls short. Temperatures fitted on one distribution should be expected to degrade on another, which is directly relevant to grouping by language or domain as well as by count. Calibration across many overlapping groups at once is studied under the name multicalibration, for example Gopalan et al. in Low-Degree Multicalibration, arXiv:2203.01255.

What the evidence says

The only decision model precedent for grouping by option count is Laya. Laya’s source file laya/common.py defines one temperature bucket per question type and option count with cut points at 2, 3 to 5, 6 to 10, and 11 and above, and Laya source clamps every applied temperature to the range 0.5 to 5.0. The code comment in that file reports the shipped choice bucket for 11 and above fitted at 0.1006, multiplying logits by about 10, and illustrates a top probability of 0.24 published as 0.99. Laya BENCHMARKS.md confirms that calibration columns in its central processing unit (CPU) sweep predate the clamp, so affected buckets are now served at 0.5 rather than 0.1006, with a rerun tracked in issue 208. Laya README reports that its fine tuning notebook now fits calibration on a held out slice rather than on training data, fixed in issue 186.

The refit effect is large. Laya BENCHMARKS.md reports mean ECE moving from 0.466 to 0.081 on the English checkpoint and from 0.314 to 0.106 on the multilingual checkpoint after refitting one temperature per question type and option count on held out data. Laya BENCHMARKS.md calls this refit the single highest value fix available, and notes that it takes ECE below the cited Jev measured figure of 0.246. Laya README adds that the multilingual checkpoint ships with no fitted temperatures at all. Miscalibration direction is not stable across tasks: Laya BENCHMARKS.md reports a router deployment over 180 requests where nearly every configuration was under confident, with chosen probability below accuracy by 0.18 and 0.19 on two checkpoints.

For flip rate, Laya BENCHMARKS.md reports option order flips of 0.150 and 0.230 on its 20 option MASSIVE intent suite for the two checkpoints, 0.040 and 0.090 on English emotion, and 0.000 and 0.015 on English Cross-lingual Natural Language Inference (XNLI), against a cited Jev measured figure of 0.13. Laya BENCHMARKS.md suggests more aggressive option order shuffling during training as the fix. Note the asymmetry: temperature grouping is the prescribed fix for ECE, while training time shuffling is the prescribed fix for flips.

Third party Jev measurements supply the closed model comparison points. The decision model benchmark README by nibzard reports Jev at 76.3 percent on its banking 77 way suite, ECE of 0.246 against a large language model (LLM) range of 0.039 to 0.122, and 13 percent of choices changing under permutation. The same README reports median (p50) latency of 264 to 276 ms flat from 2 to 255 options and refusal with 400 Too many choices at 256 and above. TypeSafe states support up to 255 options, with higher cardinality choices handled by a two stage system of independent scoring then explicit choice. The AbdelStark pilot reports Jev at 0.870 against GLiNER at 0.610 on its 72 label Banking77 BTZSC condition over 300 held out examples with 100 per condition. On 6 label DAIR Emotion the same pilot reports accuracy unresolved at 0.480 against 0.440 but worse Jev calibration: Brier score 0.846 against 0.668, negative log-likelihood (NLL) 5.588 against 1.381, and zero probability on truth for 16 percent of examples. AbdelStark notes that most Banking77 BTZSC classes have one or two sampled examples, so per class estimates are unstable, and that its pinned configuration exposes 72 hypotheses rather than 77.

The table below compares grouping strategies against this evidence.

Analysis

For fixed logits, the softmax top probability falls as competing terms are added, so a model calibrated at small counts miscalibrates at large counts unless its correction knows the count. Laya BENCHMARKS.md shows this is not second order: as shipped ECE of 0.466 and 0.314 falls to 0.081 and 0.106 after per bucket refit. Report ECE against option count as a curve, then choose buckets from the bends in that curve.

Laya buckets are the right starting shape but the top bucket is too wide. The cut points 2, 3 to 5, 6 to 10, and 11 and above come from laya/common.py, so the top bucket spans 11 options to the 255 option cap that TypeSafe states. ECE behavior at the bottom and the top of that span has no reason to share one scalar, and no fetched source measures that span. The procedure is to fit one temperature per exact count on a large held out sweep, plot fitted T and residual ECE against count, and place boundaries where the curve bends. Until that sweep exists, a reasonable interim split keeps 11 to 20 separate from above 20, since Laya BENCHMARKS.md reports behavior at 20 options. The cardinality break point design in this graph proposes the per trunk sweep that would locate such bends.

The clamp is load bearing, not cosmetic. A fitted temperature of 0.1006 on the 11 and above bucket, reported in the laya/common.py comment, sharpens logits by about 10 times, and the same comment shows a 0.24 top probability published as 0.99. Serving that value would tell a caller gating on confidence that a coin flip is a certainty. Laya clamps applied values to 0.5 to 5.0 while keeping raw values available for inspection, per laya/agent.py. Every grouped implementation should copy three properties: fit on held out data only, following the issue 186 fix that Laya README reports; clamp the applied value and log when the clamp binds; and keep the raw fit visible so a binding clamp reads as a data shortage flag.

Fix the ECE variant before comparing groupings. Nixon et al. report that the rank ordering of recalibration methods changes drastically with measure choices, so a grouping study must preregister confidence ECE with stated bins, classwise ECE, Brier score, and NLL, which are the measures Kull et al. report for Dirichlet evaluation. Post hoc selection among variants after seeing which grouping wins is not analysis. The router under confidence finding strengthens the point: Laya BENCHMARKS.md reports chosen probability below accuracy by 0.18 and 0.19 over 180 requests on one deployment, the opposite sign from the overconfidence in the main sweep.

Temperature grouping cannot move flip rate, so budget a second lever. Temperature scaling preserves each ordering argmax by construction, so the fraction of items whose top answer changes across orders is identical before and after any scalar refit on the same logits. This is why Laya BENCHMARKS.md prescribes shuffling during training for its 0.150 and 0.230 flips at 20 options rather than another temperature. The candidate levers are training time shuffling with a consistency term, which the permutation group training report in this graph takes up, and inference time prior correction in the PriDe style that Zheng et al. propose. Fitting should still run on permuted held out data so the temperature averages over orders, with ECE and flip rate measured on the same sweep and only ECE expected to respond to grouping.

Choose Dirichlet over scalar only when the held out budget supports it. Kull et al. report gains on all four of their measures, but a per bucket linear map needs roughly quadratic in class count parameters against one scalar for temperature. At 72 labels with most classes holding one or two sampled examples, which AbdelStark reports for its pilot, a Dirichlet fit is not identified. The practical ladder is: per bucket temperature first with clamp and fallback; Dirichlet per bucket only where held out samples per class suffice; scaling binning per Kumar et al. when measurable guarantees are required. The zero probability finding sharpens the data requirement: NLL of 5.588 with 16 percent exact zeros on Emotion, which AbdelStark reports, means an NLL based objective will be dominated by a few pathological items, so report zero mass items separately rather than letting them set the temperature.

Plan the fallback chain before adding grouping axes. Counts times types times languages leaves most triples empty at any realistic held out budget. The Maestro Flash proposal calls for per question type, option count, and language temperatures, which Ovadia et al. report on shift motivates, but it needs explicit precedence of full triple, then type and count bucket, then type only, then a global 1.0, which laya/common.py also uses as its no scaling fallback, with the applied level logged per decision. Laya already implements the middle of this chain: bucket specific temperatures take precedence over per type values with fallback when a bucket is absent, per Laya README. Empty bucket behavior must be specified and tested, since silently serving a global default on a slice that needed its own fit is the worst failure mode.

Implications for small decision models

For Jev, grouping must happen outside the model, because the application programming interface (API) boundary blocks every inside option. Jev internals are UNVERIFIED: no weights, training paper, or logit access are public, as the Jev survey in this graph documents. A caller cannot fit Jev temperatures or read full logits. What the caller can do is fit a per count recalibration on Jev returned probabilities, using a Dirichlet style log map or isotonic fit per count bucket on labeled held out data, since those methods need only probabilities. The decision model benchmark README reports Jev ECE of 0.246 against 0.039 to 0.122 for LLMs, so the headroom is real, but the refit inherits the 255 option cap that TypeSafe states, with refusal at 256 and above reported by the benchmark. For flips, the same benchmark reports 13 percent, and no output side scalar can reduce that; the caller side lever is permutation averaging at multiplied cost, or a PriDe style prior estimate per the Zheng et al. method.

For Laya, the grouping machinery exists and the gaps are concrete: buckets, clamp, precedence, and raw value inspection are implemented in the fetched source, and Laya BENCHMARKS.md reports the refit path from 0.466 and 0.314 to 0.081 and 0.106. The remaining work is: split the 11 and above bucket once a held out sweep locates the bend above 20 options; fit and ship multilingual temperatures, since Laya README reports none are shipped; rerun the pre clamp CPU sweep, flagged as stale with the rerun tracked in issue 208; and fit future temperatures on permuted held out data. The checkpoints are published under the convaiinnovations organization on Hugging Face, which a model search for laya returns alongside the multilingual and typed decisions checkpoints, so a corrected refit can ship as a config update. Laya’s 20-option flips of 0.150 and 0.230 stay a training problem, not a calibration problem, per the benchmarks file’s own shuffling prescription.

For Velofy’s proposed Maestro Flash, which has no weights and no results, the implication is a measurement first calibration plan. The Maestro Flash proposal calls for per question type, option count, and language temperatures with separately encoded and cacheable option representations. The evidence supports that shape with four constraints. First, preregister the ECE variant set from the Nixon et al. and Kull et al. measures before fitting anything. Second, derive bucket boundaries from a fitted T against count sweep on held out data with fixed items and only labels varying. Third, specify the fallback chain and log the applied level per decision, since most triples will be empty. Fourth, fit on permuted held out items and report ECE and flip rate on the same sweep, following the Decision Bench v0 discipline of pinned revisions. Until those curves exist, count grouped calibration for Maestro Flash is a design, not a result. This report proposes no new architecture.

Limitations

Jev internals are UNVERIFIED, and every Jev number here comes from vendor claims or two third party projects with their own prompts and samples. AbdelStark reports 300 held out examples with 100 per condition, which leaves most Banking77 BTZSC classes at one or two examples, as the same README notes. The decision model benchmark evaluates 77 way banking while AbdelStark evaluates 72 hypotheses, so the two Banking77 figures do not share a denominator. Laya figures are self reported and were read from its README, BENCHMARKS.md, and source files, not reproduced here. Cross paper ECE comparison is approximate by construction, since Nixon et al. report that measure choices drastically change method rankings. No fetched source reports a fine grained ECE against option count curve, so every boundary recommendation above is a procedure for finding bends rather than a located bend. No experiment was run for this report.

Open questions

1. At which option counts do per count ECE curves bend on fixed items, and which bucket boundaries keep residual ECE flat with the fewest held out samples per bucket?
2. Does fitting one temperature per option count bucket on permuted held out data change per bucket ECE against canonical order fitting, and can any scalar grouping move flip rate or only ECE?
3. Does per bucket Dirichlet calibration beat per bucket temperature scaling on classwise ECE and Brier at high option counts, at matched held out budget with the clamp and fallback applied?

References

1. Guo et al., On Calibration of Modern Neural Networks, arXiv:1706.04599
2. Kull et al., Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration, arXiv:1910.12656
3. Nixon et al., Measuring Calibration in Deep Learning, arXiv:1904.01685
4. Kumar et al., Verified Uncertainty Calibration, arXiv:1909.10155
5. Minderer et al., Revisiting the Calibration of Modern Neural Networks, arXiv:2106.07998
6. Ovadia et al., Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift, arXiv:1906.02530
7. Zheng et al., Large Language Models Are Not Robust Multiple Choice Selectors, arXiv:2309.03882
8. Gopalan et al., Low-Degree Multicalibration, arXiv:2203.01255
9. Laya README, NandhaKishorM/laya
10. Laya BENCHMARKS.md, NandhaKishorM/laya
11. Laya laya/common.py, temperature buckets and clamp, NandhaKishorM/laya
12. nibzard/decision-model-benchmark
13. AbdelStark/jev-benchmarks
14. TypeSafe, Introducing System One Models and Jev
15. scikit-learn documentation, Probability calibration
16. Hugging Face, convaiinnovations/laya

← All papers