Velofy Decision Bench v0: One Frozen Protocol for Small Decision Models

The question

What single, frozen benchmark protocol should measure accuracy, calibration, abstention, option-order flips, cardinality, and cost per accepted decision for small decision models? This report freezes the answer as Velofy Decision Bench v0: one suite map, one metric set, one calibration rule, one cost rule, and one reporting contract, so that two labs running it independently produce numbers comparable digit for digit.

Four sibling papers in this graph already settled the hard subproblems, and this report does not re-argue them. The flip-rate protocol fixes the permutation count (identity plus 5 seeded permutations), the six option-count buckets, and the four readouts (flip rate, mean pairwise total variation distance (TVD), permuted accuracy, permuted expected calibration error (ECE)). The unanswerable-items report fixes the three abstention families (underspecified state, contradictory options, out-of-language input), the answerable-twin design, and exact grading on an abstain flag plus a closed reason code. The cost benchmark design fixes cost per accepted decision at 1 percent and 5 percent empirical error budgets, token-class accounting at dated list rates, and three load scenarios with a utilization sweep. The stage-zero harness fixes the frozen-items contract (fixed items, fixed seeds, pinned revisions, byte-identical questions), the full-cardinality Banking77 condition, and paired bootstrap intervals for every comparison.

What remains is the freeze itself: the choices none of those papers could make alone because each owned only its axis. This report resolves the two conflicts between parents (the ECE bin count and the temperature-fit distribution), fixes suite composition and item counts, names the required baselines and the over-cap cell, defines the versioning mechanics that make “frozen” operational, and states what a v0-compliant run must print. Velofy has trained no model for this report and ships none with it. Every empirical number below comes from a fetched source and is attributed inline; anything else is marked UNVERIFIED or left out.

First principles

A benchmark number is the output of a function. Its inputs are the items, the order the options appear in, the model revision, the metric definition, the calibration state of the model and, for cost, the price list and the load. Two labs get the same number only if every one of those inputs is the same. That one fact sets the whole design of v0.

• Any input left free becomes a hidden variable. Different items give different accuracy. A different ECE bin count gives a different calibration error. If one lab refits temperatures and another does not, their two ECE figures measure different things under one name. Freezing means pinning every input and publishing the pins (hashes, seeds, revisions) so a reader can check them.
• A score on a finite item set is an estimate with an error bar. For a rate p measured on n items, the binomial standard error is the square root of p(1 - p)/n. Item counts set the smallest difference the bench can detect, so they are fixed before the run, not after.
• Averages across unlike conditions hide the failure that matters. Accuracy pooled over 2-option and 77-option items is dominated by whichever bucket is larger or easier. Per-bucket and per-family tables keep the weak cell visible, which is why v0 forbids pooling.
• Cost has a numerator and a denominator. Spend depends on tokens or machine time, price date, billing tier, and load. Value depends on how many decisions can be accepted at a stated error rate. A cost figure is comparable only when both halves are pinned.
• A floor makes every number readable. Random, majority, and keyword baselines on the same items show how much of a score the task gives away for free.

Background

Typed-decision numbers today are incomparable across labs, and the evidence for that claim is already on the record in this graph. The Jev survey documents two independent Banking77-class readings for Jev that differ in absolute terms (0.870 on a 72-label condition in one benchmark, 76.3 percent on 77-way suites in the other), which is expected because they use different items, label wordings, and protocols. The Laya teardown documents as-shipped ECE of 0.466 falling to 0.081 after a temperature refit, which means any two ECE figures measured on different sides of a refit are different quantities wearing the same name. The flip-rate parent documents Laya’s published flip table (0.150 and 0.230 at 20 options) resting on an UNVERIFIED permutation count, so no lab can rerun it exactly. Each number is honest in isolation; together they cannot be placed side by side.

The wider field has already learned, several times, what fixes this. The EleutherAI lm-evaluation-harness README describes over 60 standard academic benchmarks and states its rationale plainly: evaluation with publicly available prompts ensures reproducibility and comparability between papers [15]. Holistic Evaluation of Language Models (HELM) adopts a multi-metric approach, measuring 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency) for each of 16 core scenarios when possible, which its abstract reports as 87.5 percent of the time, across 30 prominent language models [2]. Beyond the Imitation Game benchmark (BIG-bench) reports 204 tasks contributed by 450 authors across 132 institutions [10]. Massive Multitask Language Understanding (MMLU) covers 57 tasks [3], and MMLU-Pro expands the choice set from four to ten options, reporting an accuracy drop of 16 to 33 percent and a fall in prompt sensitivity from 4-5 percent to 2 percent across 24 prompt styles [4]. MLPerf’s inference rules define four load scenarios (Single stream, Server/Interactive with Poisson arrivals, Offline, Multistream), each with runs of 600 seconds, and define throughput only against an arrival process plus a latency bound [16]. The pattern across all five is the same: freeze the items, fix the metrics, pin the load, publish everything.

Two decision-specific precedents supply the rest of the shape. The decision-model-benchmark (DMB) README reports three contender classes under one frozen protocol, five suites, 60 cells, and $28.34 of measured provider spend with every raw log published [11]. Its findings list reads like a checklist this bench inherits: Jev at $0.07 per 1,000 and 264 to 276 ms flat from 2 to 255 options, 100 percent accuracy at 254 and 255 with a 400 rejection at 256 and above, 13 percent option-permutation flips, ECE 0.246 against a large language model (LLM) range of 0.039 to 0.122, and ignorance admission on 49.7 percent of forced-uncertainty items against 97.3 to 100 percent for the LLMs [11]. The jev-benchmarks README fixes the error-budget formalism v0 adopts: 300 held-out examples, pinned revisions, paired bootstrap intervals, and coverage at 5 percent or lower error as the denominator (0.830 against 0.240 on AG News, 0.860 against 0.270 on Banking77/BTZSC, 0.000 against 0.020 on DAIR Emotion) [13]. Laya’s BENCHMARKS.md supplies the reproducibility contract (byte-identical questions per run under a fixed seed) and the calibration repair precedent (per-bucket temperature refits move ECE from 0.466 to 0.081 on English and 0.314 to 0.106 on multilingual, clamped to [0.5, 5]) [14].

What the evidence says

The table below compares the benchmark designs surveyed for this report on the six properties v0 must combine. A “yes” means the protocol pins the property; “part” means it reports the property without pinning it; “no” means it leaves the property open.

No row says yes in every column. The v0 freeze is the union of the yes cells, with the gaps filled by the four parent papers.

Four findings from the fetched sources shape the details. First, calibration repair is cheap and dominant. Guo et al. report that temperature scaling, a single-parameter variant of Platt Scaling, is surprisingly effective at calibrating predictions [1], and Laya’s BENCHMARKS.md confirms the magnitude on decision tasks: refits move ECE by 0.2 to 0.4 while the post-fit trunk gap is 0.025 [14]. Any bench that ranks calibration without a held-out refit step ranks the default temperature, not the model.

Second, option order moves answers by amounts no bench can ignore. Pezeshkpour and Hruschka report gaps of approximately 13 to 75 percent when answer options are reordered [9]. DMB measures Jev changing 13 percent of its choices under permutation [11], and Laya’s BENCHMARKS.md reports 0.150 and 0.230 on its 20-option intent suite [14]. Order robustness is a scored axis in v0, not a diagnostic.

Third, unanswerable items are a solved construction problem with measured difficulty effects. SQuAD 2.0 adds over 50,000 adversarially written unanswerable questions, and reports a strong system falling from 86 percent F1 on SQuAD 1.1 to 66 percent F1 on SQuAD 2.0 [5]. AbstentionBench evaluates 20 frontier LLMs over 20 datasets and reports that reasoning fine-tuning degrades abstention by 24 percent on average [6]. The v0 abstention slice therefore needs families and twins (per the parent report), not just volume.

Fourth, cost heterogeneity is large enough to swamp quality differences if left unpinned. FrugalGPT reports that popular LLM application programming interfaces (APIs) have heterogeneous pricing structures with fees differing by two orders of magnitude, and that a learned cascade can match the best individual LLM with up to 98 percent cost reduction or improve accuracy over GPT-4 by 4 percent at the same cost [7]. RouteLLM reports cost reductions by over 2 times in certain cases without compromising response quality [8]. A bench that prices APIs but not load, or spend but not error budgets, ranks systems on different goods.

Analysis

The v0 freeze has six parts: the suite map, the metric set, the calibration rule, the cost rule, the run rules, and the reporting and versioning contract. Each part states what is frozen, then the parent or source it inherits from.

Suite map. Six suites, all frozen at release:

The full run is 6,750 forward passes (computed). S-flip contributes 4,800 of them: 750 items at 6 orders plus the 150 binary items, which admit only 2 orders. At DMB’s measured $0.07 per 1,000 decisions for Jev on banking-style prompts [11], a full v0 run prices near $0.47 in API spend for a Jev-class contender (computed), before load-scenario repetition. The bucket boundaries and the 6-order rule come from the flip-rate parent; the families, twins, and reason codes from the unanswerable-items parent; the full-77 condition from the stage-zero parent; the over-cap cell from the cost parent. S-choice buckets carry 150 items rather than the parent’s 100 minimum so that a 0.13 flip rate [11] has a binomial standard error near 0.027 (computed).

Metric set. Every metric is micro-averaged within its bucket or family, and every row is printed. The primary figure per suite is listed first:

NLL ships wherever full distributions are logged [13]. Brier is never the sole ranking figure on any suite, because it mixes discrimination and calibration (the stage-zero parent’s rule).

Calibration rule, with two conflict resolutions. First conflict: the stage-zero parent specifies 15 ECE bins while the flip-rate parent follows the DMB SPEC definition of 10 bins. V0 uses 10-bin top-label ECE everywhere, because DMB is the closest frozen implementation and every published decision-model ECE figure (0.246 against 0.039 to 0.122 [11]) was computed under it; adopting 15 would orphan those numbers. Second conflict: the stage-zero parent fits temperatures on held-out data while the flip-rate parent requires fitting on permuted orders. V0 requires both: one temperature per (question type, option-count bucket), fit on held-out items under permuted orders, clamped to [0.5, 5.0] following Laya’s client-side clamp [14]. Post-refit ECE is the ranked figure; pre-refit ECE is printed beside it for audit. If permuted ECE exceeds unpermuted ECE by more than the bucket standard error, the report must say so: the calibration was fit on the wrong distribution.

Cost rule. For each API contender the runner records input, output, cache-hit, and cache-read tokens as metered, plus the tier of each call, and prices spend at list rates in force on the run date with the invoice total beside it as a cross-check. Retries, refusals, and malformed outputs count in full. For self-hosted contenders the runner records wall time, batch size, and concurrency, and converts to dollars at dated card rates with the billing mode stated on the label. The three required load scenarios are single-stream (p50 and p95), Poisson server (maximum sustainable rate and spend per decision at a stated latency bound), and offline throughput, each run for 600 seconds by default following MLPerf [16], with dollars per decision swept at 10, 50, and 100 percent of the maximum sustainable rate. Cold starts are stated separately, and batching delay is reported as added latency. The headline figure is cost per million accepted decisions, computed separately at 1 percent and 5 percent error; pooling across budgets is forbidden.

Run rules. Item files are hashed into the manifest; model and dataset revisions are pinned; seed lists are published; regeneration from seeds must be byte-identical. Every contender receives the same serialized decision state against the same schema, and constrained LLM baselines run at temperature 0 with native structured output where offered (both DMB SPEC rules [12]). Top-1 ties break by lowest canonical index, and count as non-flips only if the tie set is identical across orders. Malformed outputs, refusals, and over-cap rejections count in their own columns, never as wrong answers [11]. Three baselines are mandatory in every release (random, majority, keyword), so every table carries its own floors. Every comparison ships with paired bootstrap intervals over the same items [13].

Reporting and versioning. A v0-compliant run prints the full per-bucket and per-family tables with n and standard errors, the two cost-per-accepted tuples with their five labels (suite and item hash, error budget, load scenario, utilization or tier, price date), and machine-readable cells with raw counts. Three poolings are forbidden: across buckets, across budgets, and across suites. There is no single winner function; v0 ranks per axis with pre-registered vetoes (a post-refit ECE ceiling and a cap-behavior disclosure), following the stage-zero parent and HELM’s precedent of exposing trade-offs rather than collapsing them [2]. The freeze is operational: the v0 identifier and item hashes live in the manifest, audited-answerable items go into a versioned errata file, one generator per abstention family stays hidden, and any item change triggers v1. A new contender can be compared against published v0 figures without rerunning every baseline, because the items, seeds, and revisions are pinned.

Implications for small decision models

For Jev, the implication is a client-side rerun with published seeds, since Jev is a closed API known only through third-party measurements, as the Jev survey documents. DMB already ran the closest existing measurement (77-way suites, 3 permutations, 13 percent flips, ECE 0.246, $0.07 per 1,000, 400 at 256 and above [11]), so the v0 upgrade is concrete: extend to all six buckets at 6 orders per item, add the S-abstain families with twins to locate the 49.7 percent admission rate by failure kind, and convert the public calibration readings into coverage curves at both budgets. Jev’s flat 264 to 276 ms p50 [11] makes the rerun cost predictable: 6,750 passes at constant latency, with the S-cap cell expected to fail by design. Nothing here requires TypeSafe’s cooperation.

For Laya, the implication is to become v0’s first adopter, because the project already meets half the contract: byte-identical questions per run, per-bucket temperatures with the [0.5, 5] clamp, a published flip table, and openly downloadable backbones (the Hugging Face API reports 625,057 downloads for answerdotai/ModernBERT-large at fetch time [17], which makes every Laya rerun replicable by any lab). The merge list is five items: add the abstain output with the three reason codes; extend the suite set to all six buckets; report mean pairwise TVD beside flip rate; refit temperatures on permuted orders; and adopt the hidden-generator holdout. The English versus multilingual checkpoint pair is the built-in out-of-language experiment: the same held-out-language slice should show the English checkpoint abstaining and the multilingual checkpoint answering. Laya’s BENCHMARKS.md already runs MASSIVE and Cross-lingual Natural Language Inference (XNLI) across 14 and 15 languages plus a 51-language central processing unit (CPU) sweep [14], so the language menu exists; v0 only asks that part of it be graded as abstention.

For Velofy’s proposed Maestro Flash, which has no weights and no results, the implication is process. Four commitments would make the proposal v0-conformant from the first checkpoint. First, freeze the v0 item files with hashes in the run manifest before the first training run. Second, log every v0 readout per bucket from the first evaluation, and gate release notes on the 21-77 and 78-255 buckets where the incumbent single-pass marker scheme is weakest. Third, design the S-cap cell now, before the architecture freezes, since shortlist-then-decide, decoupled option encoders, and hierarchical decisions have different cost curves past any single-pass limit. Fourth, set per-family abstention operating points alongside per-bucket temperatures. The first v0 numbers should come from baselines run against the harness, not from new weights; harness first with published seeds is what lets later results be trusted.

Limitations

This report ran no experiments, trained no models, and shipped nothing; every number is a cited value from a fetched source. The item counts are principled but not derived: they balance binomial standard errors against the 6,750-pass run cost, and a lab with a different budget could freeze different counts. The 10-bin ECE choice orphans the stage-zero parent’s 15-bin specification deliberately; comparability with published decision-model figures won over bin-count purity, and a future revision with evidence that 15 bins ranks systems differently should revisit it. The LLM-side evidence studies decoder models, and only its protocol shapes transfer to encoder heads, not its magnitudes. List prices move, so the cost rule captures a single day. The public pilots are small (jev-benchmarks uses 300 items [13]), so intervals on hard slices stay wide until v0-scale runs exist. Translation-based out-of-language items risk translationese, with unmeasured audit cost. The choice suites are English-only in v0; multilingualism is measured through the abstention slice plus the MASSIVE and XNLI extension path [14]. All repository figures are snapshots as fetched in September 2026; DMB, jev-benchmarks, and Laya are fast-moving projects, and reruns may legitimately differ.

Open questions

1. How many items per bucket does v0 need for rank-stable comparisons, measured by rerunning the frozen suites with disjoint seeds and reporting rank agreement across model classes?
2. What single deployment summary of the v0 surface may a lab print (across budgets, suites, and load scenarios) without violating the no-pooling rule, so that two systems can still be ranked for one stated deployment?
3. Which hidden-generator holdout design best detects benchmark training, such as refusal-trigger memorization, without punishing legitimate fine-tuning on the public generators?

References

1. Guo et al., On Calibration of Modern Neural Networks, arXiv:1706.04599: https://arxiv.org/abs/1706.04599
2. Liang et al., Holistic Evaluation of Language Models (HELM), arXiv:2211.09110: https://arxiv.org/abs/2211.09110
3. Hendrycks et al., Measuring Massive Multitask Language Understanding, arXiv:2009.03300: https://arxiv.org/abs/2009.03300
4. Wang et al., MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark, arXiv:2406.01574: https://arxiv.org/abs/2406.01574
5. Rajpurkar et al., Know What You Don’t Know: Unanswerable Questions for SQuAD, arXiv:1806.03822: https://arxiv.org/abs/1806.03822
6. Kirichenko et al., AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions, arXiv:2506.09038: https://arxiv.org/abs/2506.09038
7. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, arXiv:2305.05176: https://arxiv.org/abs/2305.05176
8. RouteLLM: Learning to Route LLMs with Preference Data, arXiv:2406.18665: https://arxiv.org/abs/2406.18665
9. Pezeshkpour and Hruschka, Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions, arXiv:2308.11483: https://arxiv.org/abs/2308.11483
10. Srivastava et al., Beyond the Imitation Game (BIG-bench), arXiv:2206.04615: https://arxiv.org/abs/2206.04615
11. nibzard/decision-model-benchmark README: https://raw.githubusercontent.com/nibzard/decision-model-benchmark/main/README.md
12. nibzard/decision-model-benchmark SPEC.md: https://raw.githubusercontent.com/nibzard/decision-model-benchmark/main/SPEC.md
13. AbdelStark/jev-benchmarks README: https://raw.githubusercontent.com/AbdelStark/jev-benchmarks/main/README.md
14. NandhaKishorM/laya BENCHMARKS.md: https://raw.githubusercontent.com/NandhaKishorM/laya/main/BENCHMARKS.md
15. EleutherAI lm-evaluation-harness README: https://github.com/EleutherAI/lm-evaluation-harness
16. MLPerf inference rules: https://raw.githubusercontent.com/mlcommons/inference_policies/master/inference_rules.adoc
17. Hugging Face model API, answerdotai/ModernBERT-large download count fetched for this report: https://huggingface.co/api/models?search=ModernBERT&limit=3

← All papers