The option-count sweep left three open questions. This report answers all three with new measurements from the same local harness, the same 150 Banking77 test items, and the same three systems: a trained term frequency-inverse document frequency (TF-IDF) plus logistic regression (LR) baseline, Laya zero-shot, and GLiClass-small zero-shot. Nothing was fit on the test items, the abstention threshold stays at a fixed 0.5, and the calibration pool shares no text with the test slice. The run took 1,083 s on an Apple M3 Pro on the central processing unit (CPU).
First principles
The three experiments rest on four plain facts about how a decision pipeline is built.
- An item’s difficulty depends on the options shown, not only on the question. A zero-shot model scores each option by reading its text against the state. A wrong option that shares words and meaning with the right one is harder to reject than an unrelated one. Accuracy measured against random distractors is therefore a best case for the confusable options real applications present.
- Temperature changes sharpness, never the top choice. Dividing every logit by one positive number keeps their order. Temperature can make confidence honest. It cannot make the model better at telling right answers from wrong ones.
- One parameter fit to pooled data fits the average. If the right temperature differs between 5 and 77 options, a single temperature fit across both is a compromise that suits neither. Fitting per bucket is the direct remedy, and the data show whether it is needed.
- A two-stage pipeline is capped by its first stage. The decider can only pick from the shortlist, so pipeline accuracy can never exceed shortlist recall. The second stage adds value only when it ranks the shortlist better than the first stage’s own top choice.
Experiment C: near-miss distractors
The sweep used random distractors, which are easy to reject. Here each item’s k-1 distractors are the labels closest to its true label, by cosine similarity of label-name embeddings from sentence-transformers/all-MiniLM-L6-v2. For example, a card-arrival question now competes with other card-delivery labels instead of exchange rates.
| System | k | Accuracy, near-miss / random | Flip rate, near-miss / random | Coverage at 5% error, near-miss / random |
|---|---|---|---|---|
| TF-IDF + LR | 5 | 0.927 / 0.980 | 0.000 / 0.000 | 0.947 / 1.000 |
| TF-IDF + LR | 20 | 0.893 / 0.913 | 0.000 / 0.000 | 0.880 / 0.947 |
| Laya | 5 | 0.713 / 0.887 | 0.167 / 0.067 | 0.287 / 0.900 |
| Laya | 20 | 0.520 / 0.727 | 0.313 / 0.167 | 0.007 / 0.047 |
| GLiClass-small | 5 | 0.593 / 0.740 | 0.187 / 0.160 | 0.000 / 0.160 |
| GLiClass-small | 20 | 0.300 / 0.333 | 0.393 / 0.340 | 0.013 / 0.000 |
The sweep’s most favorable result for zero-shot models was Laya at 5 random options: 0.887 accuracy, 6.7% flips, and 90% of items answerable at a 5% error budget. With near-miss options, accuracy drops to 0.713, flips more than double, and selective coverage falls from 0.900 to 0.287. Most of that small-k advantage came from easy distractors. Real deployments rarely offer random alternatives. Their options tend to be the confusable ones, so near-miss numbers are the better guide.
The trained baseline barely moves (0.980 to 0.927 at k=5) and never flips, because it has seen every label during training and never reads the option list.
Experiment D: one temperature per option-count bucket
The sweep refit used one global temperature. Here, each zero-shot system gets one temperature per option count (5, 20, 77), fit on 600 train items per bucket, and is compared against a single temperature fit on all buckets pooled.
| System | T at k=5 | T at k=20 | T at k=77 | Global T |
|---|---|---|---|---|
| Laya | 1.107 | 3.864 | 3.715 | 2.418 |
| GLiClass-small | 1.548 | 2.325 | 3.832 | 2.603 |
The fitted temperatures vary across buckets by 3.5 times for Laya and 2.5 times for GLiClass-small. A single global value is therefore a compromise that fits no bucket: too soft at 5 options and too sharp at 77.
On the test cells, per-bucket temperatures beat the global one on negative log-likelihood (NLL) in all 6 system-bucket cells, and are better or equal on Brier in all 6 (GLiClass-small at 77 options ties at 0.977). They beat it on expected calibration error (ECE) in 5 of 6. The exception is GLiClass-small at 77 options, where global ECE is 0.025 against per-bucket 0.047. For Laya at 5 options, the clearest case, per-bucket ECE is 0.048 against 0.232 global.
Two caveats keep this from being a clean win. First, for Laya at 5 options the unscaled model (T=1) is already best: ECE 0.031 and NLL 0.315, against 0.048 and 0.320 per-bucket. Its fitted temperature of 1.107 barely moves it, and the global fit makes it worse. Second, temperature cannot create ranking signal. Selective coverage at a 5% error budget stays within one step of its unscaled value in every cell, and GLiClass-small at 20 and 77 options abstains on everything at 0.5 under any fitted temperature. Per-bucket temperature is the right calibration default for zero-shot decision models. It is not a fix for weak ranking.
This measurement supports the recommendation in calibration grouping by option count and calibrating sigmoid label scorers: fit calibration per option-count bucket, never pooled.
Experiment E: shortlist, then decide
The trained baseline produces a top-m shortlist from all 77 labels, and Laya then decides among the shortlist.
| Setup | Accuracy | Shortlist recall (misses) | Flip rate | Median (p50) latency |
|---|---|---|---|---|
| Top-3 shortlist, then Laya | 0.747 | 0.947 (8) | 0.093 | 85.3 ms |
| Top-5 shortlist, then Laya | 0.687 | 0.967 (5) | 0.140 | 105.3 ms |
| Top-10 shortlist, then Laya | 0.627 | 0.993 (1) | 0.227 | 145.9 ms |
| TF-IDF + LR alone, 77 options | 0.867 | 0.000 (order-invariant) | 0.2 ms | |
| Laya alone, 77 options | 0.333 | 0.553 | 272.7 ms |
A shortlist more than doubles Laya’s accuracy at 77 labels (0.333 to 0.747), cuts its flip rate from 55.3% to 9.3%, and roughly triples its speed, because it now scores 3 options instead of 77. A smaller shortlist is better on every axis except recall, which is the pattern the shortlist rerank protocol paper predicted.
The shortlist still loses to the shortlister. The trained baseline alone scores 0.867 in 0.2 ms. When the true label is on the top-3 shortlist, Laya picks it 78.9% of the time. Since the pipeline (0.747) scores below the baseline’s own top-1 answer (0.867), Laya on net overrules more correct baseline answers than it rescues wrong ones. On a task with training data, adding a zero-shot decider after a trained ranker makes the pipeline worse. A shortlist is worth having only where the second stage knows something the first stage does not, such as a fine-tuned decider or information outside the label text.
What these experiments support
On this task and hardware:
- Evaluate typed-decision models on near-miss options. Random distractors overstate zero-shot accuracy and selective coverage by a wide margin at small option counts.
- Calibrate per option-count bucket. The best temperature varies by 2.5 to 3.5 times across buckets, and pooling is worse on almost every metric. Temperature still cannot rescue a model whose confidence does not rank its errors.
- Shortlisting fixes a zero-shot decider’s order sensitivity and cost at high cardinality. But a zero-shot second stage should not be placed after a stronger trained first stage.
These runs say nothing about fine-tuned Laya or GLiClass, larger checkpoints, other tasks, or any Velofy model. They also leave out TypeSafe Jev, a closed application programming interface (API) we did not test. Velofy has not trained a model. The harness is local and not yet public.
Deviations
- 150 items, two or three option counts, and three orders per cell, time-boxed for a local CPU run.
- Near-miss neighbors come from label names only, with one embedding model.
- One calibration pool of 600 train items per bucket. The v0 spec’s permuted held-out calibration set is larger.
- At k=77, unanswerable items show 76 options, because the true label is removed.
- Coverage thresholds are realized in-sample, which is optimistic for every system equally.
Open questions
- Does a fine-tuned Laya checkpoint keep its accuracy under near-miss distractors, or does fine-tuning mostly teach it the easy separations?
- When the second stage is fine-tuned on the same train split, does shortlist-then-decide beat the trained ranker alone?
- Is there a zero-shot confidence signal, such as the margin between the top two options or agreement across orders, that ranks errors at 20 options where temperature cannot?