The first Decision Bench run ended with three open questions. This report answers two of them with new measurements from the same harness, machine, and Banking77 test slice: does a post-hoc temperature refit fix calibration, and how do accuracy, order flips, and selective coverage change as the option set grows? Nothing was fit on the test slice, the abstention threshold stays at a fixed 0.5, and train and test share no text. Both experiments ran on an Apple M3 Pro on the central processing unit (CPU), with the refit taking 948 s and the sweep 528 s.
First principles
Each system here turns an item into one score per option, and those scores are normalized into probabilities over the options shown. Five facts about that pipeline decide what the two experiments can and cannot show.
- Temperature changes sharpness, not order. Dividing every logit by the same positive number keeps their ranking. Accuracy cannot move. A temperature below 1 sharpens the probabilities and one above 1 softens them, so a refit can only change how confident the model sounds.
- A fixed threshold reads that confidence. With abstention at 0.5, rescaling moves items across the line. A refit therefore trades false abstentions against caught unanswerable items. It helps only if the new confidences match how often the model is right.
- More options means more ways to be wrong. Chance accuracy is 1/k: 20% at 5 options and about 1.3% at 77. Each added distractor is one more option that could outscore the right one.
- Order sensitivity depends on what the model reads. A zero-shot model reads the option list, so where an option sits can change its score. A trained classifier scores a fixed label set and never reads the list, so it is order-invariant by construction.
- Selective coverage needs ranking, not just accuracy. Answering only the confident items works only if confidence puts right answers above wrong ones. A model can be fairly accurate and still have confidence that does not separate the two.
Experiment A: temperature refit
The Banking77 train split was divided 90/10 with a fixed seed. A term frequency-inverse document frequency (TF-IDF) plus logistic regression (LR) model was fit on the 90%, and a single temperature on the 10% by minimizing negative log-likelihood (NLL). For Laya, whose probabilities were already produced, a temperature was fit on a train subsample of at most 1,000 items, disjoint from the test slice. Temperature scaling divides the logits by one number, so it cannot change which label a model picks. Accuracy is identical before and after by construction. ECE10 below is expected calibration error (ECE) over 10 confidence bins.
| System | Fitted T | ECE10 | Brier | NLL | Coverage at 5% error | False abstentions at 0.5 | Abstention recall at 0.5 |
|---|---|---|---|---|---|---|---|
| TF-IDF + LR, before | 0.466 | 0.494 | 1.377 | 0.623 | 0.691 | 1.000 | |
| TF-IDF + LR, after | 0.332 | 0.020 | 0.222 | 0.576 | 0.781 | 0.107 | 0.631 |
| Laya, before | 0.500 | 1.080 | 9.691 | 0.001 | 0.073 | 0.131 | |
| Laya, after | 3.785 | 0.279 | 0.887 | 8.963 | 0.001 | 0.389 | 0.643 |
Both models had 0.840 (TF-IDF, trained on 90% of train) and 0.386 (Laya) accuracy before and after. This baseline differs slightly from the first run’s 0.846 because it was fit on less data.
For the trained baseline, the refit does what the calibration literature predicts. A temperature below 1 sharpens an underconfident model, ECE falls from 0.466 to 0.020, and the model becomes usable at a fixed threshold: it now wrongly abstains on 10.7% of answerable items instead of 69.1%. Selective coverage at a 5% error budget rises from 0.623 to 0.781, because the rescaled confidences separate right and wrong answers better across items. The cost is that it now catches 63.1% of unanswerable items rather than all of them. The old “perfect” recall came from refusing almost everything.
For Laya zero-shot at 77 options, a temperature above 1 softens an overconfident model and roughly halves ECE, from 0.500 to 0.279. It does not create usable selective prediction: coverage at a 5% error budget stays at 1 item in 1,001. The refit also moves the operating point rather than improving it. Abstention recall rises from 0.131 to 0.643, but false abstentions rise from 0.073 to 0.389. A single temperature cannot give a 77-way zero-shot ranking the ranking signal it lacks.
Experiment B: the option-count sweep
150 test items were drawn with a fixed seed. For k in 5, 20, and 77, each item’s option set was its true label plus k-1 distractors sampled at random with a fixed seed, and every system saw identical option sets. Each cell was run in three option orders. Mean pairwise total variation distance (TVD) measures how far the probability vectors from different orders sit from each other. TF-IDF scores all 77 labels, so its probabilities were restricted to the shown options and renormalized. Distractors were random rather than chosen to be similar, so small-k cells are easier than a real application’s near-miss options would be.
| System | k | Accuracy | Flip rate | Mean pairwise TVD | Coverage at 5% error |
|---|---|---|---|---|---|
| TF-IDF + LR | 5 | 0.980 | 0.000 | 0.000 | 1.000 |
| TF-IDF + LR | 20 | 0.913 | 0.000 | 0.000 | 0.947 |
| TF-IDF + LR | 77 | 0.867 | 0.000 | 0.000 | 0.407 |
| Laya | 5 | 0.887 | 0.067 | 0.056 | 0.900 |
| Laya | 20 | 0.727 | 0.167 | 0.117 | 0.047 |
| Laya | 77 | 0.333 | 0.553 | 0.417 | 0.020 |
| GLiClass-small | 5 | 0.740 | 0.160 | 0.115 | 0.160 |
| GLiClass-small | 20 | 0.333 | 0.340 | 0.173 | 0.000 |
| GLiClass-small | 77 | 0.093 | 0.727 | 0.202 | 0.000 |
The sweep changes how the first run should be read. At 5 options, Laya zero-shot is accurate (0.887), mostly order-stable (6.7% flips), and its confidence supports answering 90% of items at a 5% error budget. That fits what Laya was designed for: a handful of typed options per question. The first run tested it at 77 options, the regime its own documentation flags as a weakness. From 5 to 77 options its accuracy falls from 0.887 to 0.333, its flip rate rises more than eightfold, and its selective coverage collapses. The prediction in the cardinality break-point paper is that zero-shot typed-decision models degrade sharply as option count grows. This sweep is consistent with that, for two open models, on one task, at three points.
GLiClass-small follows the same shape from a lower starting point. The trained baseline loses accuracy more slowly (0.980 to 0.867) and never flips, because options never enter its input. Its selective coverage still falls at 77 options, to 0.407 on this 150-item sample (the first run measured 0.644 on the full 1,001 items, and thresholds realized on small samples are noisy). High cardinality strains confidence even for a trained model.
What these experiments support
On this task and this hardware, the measurements support three claims:
- A train-only temperature refit is a cheap and effective fix for a trained model’s calibration. It is not a fix for a zero-shot model whose confidence does not rank its errors.
- Option count is a first-order variable for zero-shot decision models. Results at one k do not transfer to another, and any benchmark that reports a single k hides the most important curve.
- Order stability and selective coverage degrade together as k grows. The standard flip-rate protocol should report them per option-count bucket, which Decision Bench v0 already requires.
The measurements do not cover fine-tuned models, larger GLiClass checkpoints, similar-distractor option sets, or any Velofy model. They also leave out TypeSafe Jev, a closed application programming interface (API) we did not test. Velofy has not trained a model. The harness remains local and unpublished for now.
Deviations
- The refit TF-IDF model was trained on 90% of the train split, so its before-refit numbers differ slightly from the first run.
- Laya’s temperature was fit on at most 1,000 train items, not a full held-out permuted calibration set as the v0 spec prescribes.
- The sweep uses 150 items, three option counts, and three orders, with random rather than adversarial distractors.
- A single global temperature per system, not per option-count bucket.
- Cost and latency were not re-reported. See the first run.
Open questions
- With distractors chosen as the nearest labels in embedding space, how much of Laya’s small-k advantage survives?
- Does fitting one temperature per option-count bucket recover any selective coverage for zero-shot models at 20 options, where Laya is still reasonably accurate?
- At what option count does a shortlist step (the trained baseline’s top 5, then Laya) beat either system alone on accuracy and flip rate?