This is the first time the Velofy graph reports numbers we measured ourselves. We ran a reference harness for Decision Bench v0 on one Mac, on a public dataset, against three systems. The run is a deliberately small subset of the v0 protocol, and every departure from the spec is listed below. Nothing was tuned on the test split. The trained baseline was fit on the train split only, and no other system saw Banking77 labels.
The short version: on a closed 77-label intent task where training data exists, a term frequency-inverse document frequency (TF-IDF) plus logistic regression model that answers in 0.19 ms beats both zero-shot decision models by a wide margin, and it is order-invariant by construction. Neither zero-shot model produced confidence that could support selective answering at this label count. That is a statement about zero-shot use at 77 options, not about what these models do after fine-tuning.
First principles
A typed decision over 77 labels has three parts: the input text, the label set, and a probability for each label. Each system in this run gets from the first two to the third in a different way, and that difference predicts most of the results below.
- Chance is low at 77 options. A uniform guess is right 1 time in 77, about 0.013, which is where both floors sit. Any number well above that is information the system brought with it.
- Examples carry more than names. The trained baseline sees 10,003 labelled questions, about 130 per label on average. A zero-shot model sees only the label name, a short phrase such as “card arrival”. It must match a customer’s question to that phrase using only what it learned in pretraining.
- A model that never reads the option list cannot depend on its order. The trained baseline maps text to a fixed output layer, so reordering the presented options cannot change its answer. Models that read the options as part of their input can be moved by order, and more options give order more room to matter.
- Accuracy and calibration are separate properties. Accuracy asks whether the top answer is right. Calibration asks whether a stated 0.7 is right about 70% of the time. A model can be accurate and underconfident, or inaccurate and honestly unsure, so both are measured.
- Abstention and selective answering only work if confidence ranks items. A threshold can only separate right answers from wrong ones when right answers carry higher confidence. If scores are nearly flat, no threshold helps.
- Local cost is time multiplied by a rate. With no metered price, cost per decision is wall time times an assumed hourly rate, so it moves with the hardware and the rate chosen.
Setup
- Task. A stratified 1,001-item test slice of Banking77 (77 intent labels), seed 20260923, dataset revision
90d4e2ee. Label text is the canonical label id with underscores replaced by spaces, identical for every system. - Systems.
- TF-IDF plus logistic regression (LR), fit locally on the 10,003-item train split (7.3 s). This is the bar a decision model has to clear when labelled data exists.
- Laya (
convaiinnovations/laya, package 0.3.7), zero-shot, one forward pass over all 77 options. - GLiClass-small (
knowledgator/gliclass-small-v1.0, 144M parameters, package 0.1.20), zero-shot, through its official single-label pipeline.
- Floors. A seeded random guesser (0.014) and the majority class (0.013).
- Hardware. Apple M3 Pro, macOS 26.3, torch 2.14.0 on the central processing unit (CPU), single stream, batch size 1, after warmup. The full run took 542.5 s.
Accuracy and calibration
Accuracy carries a 95% confidence interval (CI). Calibration is reported three ways: expected calibration error (ECE) over 10 bins, Brier score, and negative log-likelihood (NLL). Lower is better for all three.
| System | Accuracy (95% CI) | ECE, 10 bins | Brier | NLL |
|---|---|---|---|---|
| TF-IDF + LR, trained | 0.846 (0.824 to 0.868) | 0.451 | 0.471 | 1.309 |
| Laya, zero-shot | 0.386 (0.357 to 0.414) | 0.500 | 1.080 | 9.691 |
| GLiClass-small, zero-shot | 0.119 (0.100 to 0.139) | 0.127 | 0.998 | 5.312 |
The trained baseline is accurate but badly underconfident. A 77-way logistic regression spreads probability mass thinly, so its top probability is usually well below its actual hit rate. That is what an ECE of 0.451 at 0.846 accuracy means, and it is exactly the case the post-hoc refit in the v0 spec exists to fix. This run did not refit.
Laya’s 0.386 sits just below the 0.425 Banking77 figure in its own benchmark document, and its documentation is explicit that base checkpoints are near chance zero-shot and that its capability comes from fine-tuning. Two measurement notes apply to Laya. Its checkpoint ships a choice:11+ temperature of 0.1006, outside the package’s allowed range, so the package clamps it to 0.5 and warns that confidence at this option count is uncalibrated. The warning fired during our run. Laya also returns probabilities rounded to four decimals, so true-label mass below 0.00005 becomes 0.0, which inflates its NLL. We report the number as measured.
GLiClass-small’s low ECE is not good calibration. Its per-label sigmoid scores, renormalized over 77 labels, are nearly flat, so its stated confidence is low and so is its accuracy. We checked the harness here specifically: on a 5-option sanity query the pipeline’s raw scores and the harness’s parsed scores match exactly, and the model picks the right label with 0.93 confidence. The 0.119 is the model’s real zero-shot result at 77 labels, for this small v1.0 checkpoint. Newer and larger GLiClass checkpoints exist and were not tested.
Option order
On a shared 154-item subsample, each system saw the options in three orders: the original plus two seeded shuffles. Flip rate is the share of items whose top answer changes across orders. Mean pairwise total variation distance (TVD) measures how far the full probability distribution moves, even when the top answer stays.
| System | Flip rate | Mean pairwise TVD | Accuracy under permutation |
|---|---|---|---|
| TF-IDF + LR | 0.000 | 0.000 | 0.877 |
| Laya | 0.565 | 0.426 | 0.355 |
| GLiClass-small | 0.740 | 0.208 | 0.134 |
TF-IDF never sees the option list, so 0.000 is the analytic result, measured through the same loop as everything else. Laya changes its top answer on 56.5% of items when the 77 options are reordered. Laya’s own benchmark reports 0.150 at 20 options, and the flip-rate protocol and cardinality papers in this graph predicted that order sensitivity grows with option count. This is one measurement consistent with that prediction, at K=3 orders, not a confirmation of it. GLiClass-small flips more often but moves less probability mass per flip, which is what near-flat scores produce.
Abstention
Seven labels were held out of the presented option set, leaving 70 options, so 84 items had no correct answer on screen. The right behavior is to abstain. Each held-out item has a twin with the full 77 options. All systems abstain when their top probability is below a fixed 0.5, chosen in advance and not tuned.
| System | Abstention recall (unanswerable) | False abstentions (answerable) | Twin accuracy |
|---|---|---|---|
| TF-IDF + LR | 1.000 | 0.669 | 0.845 |
| Laya | 0.131 | 0.073 | 0.500 |
| GLiClass-small | 1.000 | 1.000 | 0.167 |
Read the first two columns together. A perfect abstention recall means little if the system also refuses answerable questions. GLiClass-small abstains on everything, and the underconfident TF-IDF model abstains on 67% of answerable items at this threshold. Laya does the opposite: it answers almost everything, including 87% of the questions whose correct label was not offered. No system here abstains well at a fixed threshold. That is the gap the abstention and selection head papers are about.
Selective prediction tells the same story. Holding empirical error to 5%, the TF-IDF model can answer 64.4% of items. Both zero-shot systems collapse to 1 item in 1,001, because their confidence carries no usable ranking signal at this label count. These thresholds were realized on the evaluation slice itself, which makes them optimistic.
Latency and cost
Latency is reported as the median (p50) and the 95th percentile (p95).
| System | p50 | p95 | Local cost per 1,000 decisions |
|---|---|---|---|
| TF-IDF + LR | 0.19 ms | 0.21 ms | $0.00003 |
| Laya | 278.6 ms | 325.4 ms | $0.040 |
| GLiClass-small | 64.5 ms | 75.3 ms | $0.0092 |
Cost is mean wall time per decision times an assumed $0.50 per hour for local hardware. That rate is an assumption, not a metered price, and power and depreciation are not measured. This build of torch ran on the CPU. The Laya teardown measured 79.6 ms per 3-question call on Apple Silicon Metal Performance Shaders (MPS), so Laya’s latency here reflects CPU inference over 77 options, not its best case.
Deviations from the v0 protocol
- One 77-way bucket only. The spec defines six suites (choice, flip, ordinal, boolean, abstain, over-cap).
- Three option orders on 154 items, not six orders on the full slice.
- One unanswerable construction (held-out labels, 84 items), not three generator families with reason codes.
- No temperature refit. All calibration numbers are as shipped.
- Abstention is a threshold on top probability, not a native abstain output.
- Random and majority floors only. The spec’s keyword baseline and paired bootstrap comparisons are missing.
- Cost is local wall time at an assumed rate, with no token accounting or load scenarios.
- No over-cap cell and no errata file.
What this run supports, and what it does not
It supports three claims. On a closed label set with training data, a trained linear baseline is the bar to beat, and zero-shot decision models do not clear it at 77 labels. Option-order sensitivity at high cardinality is large for the open zero-shot models we tested. And abstention at a fixed threshold is not solved by any system here, trained or not.
It does not support any claim about fine-tuned Laya or GLiClass, about larger GLiClass checkpoints, about TypeSafe Jev (not tested; it is a closed application programming interface (API)), or about any Velofy model. Velofy has not trained a model. The harness is a local reference implementation and is not yet public; we intend to publish it with the full v0 protocol.
Open questions
- Does a post-hoc temperature refit on held-out permuted data close the TF-IDF baseline’s calibration gap without hurting its selective coverage?
- After fine-tuning on the train split, where do Laya and a current GLiClass checkpoint land on accuracy and flip rate relative to the trained linear baseline?
- How do flip rate and selective coverage change for each system as the option set grows from 5 to 20 to 77 on the same items?