Fixing the Denominator: A Cost-per-Accepted-Decision Benchmark Design

The question

This report answers the first open question of decision economics: what would a cost-per-accepted-decision benchmark need to fix, across prompt shapes, token accounting, error budgets, and serving utilization, so that one dollars-per-million figure is comparable across commercial application programming interfaces (APIs), rented graphics processing units (GPUs), and local hardware. The parent report showed that headline cost figures swing by an order of magnitude when any of those four factors moves. This report does not repeat its price tables. It specifies the measurement design that would make the next such table honest.

The proposed headline metric is cost per accepted decision: total measured spend divided by the number of decisions the system accepts at a fixed empirical error budget, scaled to one million. Spend is measured money out. Acceptance is threshold-realizable coverage at the stated budget, the definition used by AbdelStark/jev-benchmarks. The benchmark makes that ratio reproducible: same suite, same budget, same load, comparable dollars.

Every number below comes from a source fetched for this report on 2026-09-23 and is attributed inline. Computed figures show their arithmetic. Anything not verified is marked UNVERIFIED. Nothing here claims Velofy has trained or shipped any model.

First principles

Cost per accepted decision is a ratio, so it can go wrong in two ways: a numerator that measures the wrong spend, or a denominator that counts the wrong decisions. Five facts decide both.

• Spend is either tokens times price or time times rate. An API bills tokens by class (input, output, cached) at a price that depends on the tier. A rented or owned machine bills time. The two only meet once machine time is divided by the decisions it served.
• Utilization turns time into cost per decision. A machine billed by the hour costs the same whether it serves one request or a thousand. At 10 percent utilization each decision costs ten times what it costs at full load. Per-second billing that scales to zero removes idle cost but adds cold starts. A self-hosted cost figure without a load shape and a utilization level is incomplete.
• Prompt shape sets the token count. A typed decision sends the state and every option label, and gets a short answer back. Input tokens dominate and grow with option count, so a figure measured on 4-option prompts says little about 77-option prompts.
• A decision is worth paying for only if it can be accepted. A system that is right only when confident must hand the rest to a fallback. The useful count is the decisions that clear a fixed error budget, which depends on calibration as much as accuracy. Dividing spend by that count, not by attempts, puts accuracy, calibration, and price on one axis.
• Failed attempts are still spend. Retries, refusals, and over-cap rejections cost money and produce no decision, so they belong in the numerator.

Each rule in the design below follows from one of these five points.

Background

Three pricing regimes cover nearly all typed-decision deployments, and the parent report documents each with fetched list prices. This section states only what the design must handle.

Per-token APIs charge for input and output tokens at list rates, with discounts that change the effective price by multiples. Anthropic documents saving 50 percent with batch processing. DeepSeek documents off-peak rates at half of peak and cache-hit input at $0.006 per million against $0.30 cache-miss at peak for its flash tier. Together AI lists separate batch API prices and cached-input prices per model, for example MiniMax M3 at $0.30 input, $0.06 cached input, and $1.20 output per million tokens. A benchmark that prices API spend must record which tier each call used.

Rented GPUs charge for time, not tokens. Modal bills per second, listing T4 at $0.000164 and H100 at $0.001097 per second, and states that the buyer never pays for idle resources, only actual compute time. Lambda lists hourly instance prices including 1x H100 SXM at $4.29 and 8x H100 SXM at $3.99. The same throughput on per-second billing with scale to zero and on a reserved hourly instance produces different dollars per decision at every utilization below 100 percent. A benchmark that compares rented GPUs must fix the load shape and the utilization, not just name the card.

Local hardware has sunk cost and near-zero marginal cost per decision, but a fixed throughput ceiling. AbdelStark/jev-benchmarks reports GLiNER2.5 near 44 ms median (p50) latency on an Apple M4 Max central processing unit (CPU) against hosted Jev at 236 to 256 ms from France. Local figures are meaningless without hardware, threading, and batch shape attached.

The academic starting points are three. FrugalGPT (arXiv:2305.05176) reports that popular large language model (LLM) APIs have heterogeneous pricing structures with fees differing by two orders of magnitude, and that a learned cascade can match the best individual LLM with up to 98 percent cost reduction or improve accuracy over GPT-4 by 4 percent at the same cost. RouteLLM (arXiv:2406.18665) reports that learned routers between a stronger and a weaker model reduce costs by over 2 times in certain cases without compromising response quality. Holistic Evaluation of Language Models (HELM, arXiv:2211.09110) measures 7 metrics including efficiency for each scenario, and reports raising shared scenario coverage across 30 models from an average of 17.9 percent to 96.0 percent under standardized conditions. Cost, routing, and standardization are joint problems: a benchmark that fixes prompts but not load, or spend but not error budgets, ranks systems on different goods.

What the evidence says

Six existing efforts each fix part of the problem. None fixes all four factors, which is why the design in the next section borrows one piece from each.

HELM fixes scenarios and metrics but prices nothing. Its abstract describes a multi-metric approach over accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency, with targeted evaluations for specific aspects. Efficiency as a first-class metric is the precedent for refusing accuracy-only rankings, but HELM reports no dollars and does not separate API spend from self-hosted time.

MLPerf inference rules fix load scenarios and quality gates but not money. The fetched rules define four scenarios: Single stream, where the load generator sends the next query when the system completes the previous one; Server, where queries arrive on a Poisson distribution and the metric is the maximum supported Poisson throughput parameter; Offline, where all samples arrive at once and the metric is measured throughput; and Multistream. Runs last 600 seconds with benchmark-specific latency constraints, plus time-to-first-token and time-per-output-token constraints for LLM tasks. Throughput is only defined against an arrival process plus a latency bound. A dollars-per-decision figure without those two is undefined for every self-hosted contender.

Artificial Analysis fixes a token-accounting convention for APIs. Its methodology page defines time to first token (TTFT), output tokens per second (tok/s), and time to 100 output tokens, and prices models at a blended rate assuming a 7:2:1 ratio of cache-hit, input, and output tokens. Publishing the blend ratio is the honest part: any reader can rescale it. The weakness for typed decisions is that the ratio is tuned for chat, where output tokens dominate cost, while a typed decision is input heavy with a short structured answer.

The nibzard decision-model benchmark fixes a frozen protocol with measured spend. Its README reports three contender classes under one frozen protocol, five suites, 60 cells, $28.34 of measured provider spend with every raw log published, and a v1 grid cost of $28.14 against a $40 soft cap and a $60 hard cap enforced by the runner. Its headline table reports cost per 1,000 on the banking suite alongside p50 latency, with Jev at $0.07 and 264 to 276 ms flat from 2 to 255 options, against LLM rows from $0.19 to $2.48. Two protocol details matter: cost cells whose run hit a retry are lower bounds marked with a dagger, and suite items rebuild byte-identical from a pinned seed with published hashes. The benchmark also documents sharp edges a cost design must price: Jev returns 400 Too many choices at 256 and above, flips 13 percent of choices under option permutation, and admits ignorance on only 49.7 percent of forced-uncertainty items against 97.3 to 100 percent for the LLMs, with expected calibration error (ECE) 0.246 against an LLM range of 0.039 to 0.122.

AbdelStark/jev-benchmarks fixes the error-budget formalism. It evaluates Jev against fastino/gliner2.5-multi-v1 on 300 held-out examples, 100 per condition, with fixed revisions, identical examples, and paired bootstrap intervals. Its pilot table reports coverage at 5 percent or lower error of 0.830 against 0.240 on AG News, 0.860 against 0.270 on Banking77/BTZSC, and 0.000 against 0.020 on DAIR Emotion. On the emotion condition Jev is substantially worse calibrated, with Brier score 0.846 against 0.668, negative log-likelihood (NLL) 5.588 against 1.381, and zero probability on the true label for 16 percent of examples. Its systems columns record p50 and 95th-percentile (p95) latency, input tokens, failures, and resolved model identity. This is the closest public precedent for the accepted-decision denominator.

Laya’s public benchmarks fix reproducibility and overhead decomposition for one self-hosted system. Its BENCHMARKS.md states that every checkpoint answered byte-identical questions per run, and that its Jev figures are third-party published and never measured there, so sample sizes and prompts differ. Its T4 table runs from 39.5 ms for one English question to 337.4 ms for 50 multilingual questions, summarizing batched throughput as 103 to 332 questions per second on a single T4, roughly 6 to 7 times faster than Jev’s independently measured 236 to 276 ms p50. Its README adds 33 ms for one question against 7.2 ms per question batched, and cold reload medians of 7.4 s on CPU and 10.3 s on T4. Two decompositions matter below: on a GB10, roughly 93 ms per call is fixed overhead with each extra question costing about 7.0 ms, and on a laptop CPU, thread pinning alone moved a three-question call from 9,396 ms to 783 ms.

Serving systems research fixes the mechanism by which utilization moves cost. The PagedAttention paper (arXiv:2309.06180) reports that vLLM improves throughput of popular LLMs by 2 to 4 times at the same latency against systems such as FasterTransformer and Orca, by cutting key-value (KV) cache waste and sharing cache within and across requests. The vLLM README lists the operative knobs: continuous batching of incoming requests, chunked prefill, and prefix caching. The encoder analogues are batching questions into one forward pass and the single-pass structure itself, which the GLiClass README reports as approximately 10 times faster than cross-encoder models, whose abstract notes they must process text-label pairs sequentially (arXiv:2508.07662). Any benchmark that lets one contender batch and another run serially, without recording it, measures the harness, not the models.

The comparison below summarizes what each effort fixes. A check means the factor is pinned by the protocol; a half mark means it is reported but not pinned; empty means it is left open.

No row checks all five columns. The design below is the union of the checked cells, adapted to typed decisions.

Analysis

The benchmark pins four factors and one reporting rule. Each choice names what the runner records so a later reader can rescale the headline figure.

Prompt shapes: freeze stratified suites with hashed items and an explicit over-cap cell. Option count is a cost dial on every token-priced API, because every option label appears in the request, and an accuracy dial on joint encoders, because option text shares the context budget. The suite must therefore stratify by option count: binary and small (2 to 6), medium (around 20), high (72 to 77, the Banking77 shape), cap edge (254 to 255), and over cap (256 and above). State passages need fixed length bands as well, since long states inflate input tokens for APIs while crowding out option text for joint encoders. Items rebuild byte-identical from a pinned seed with published hashes, and model and dataset revisions are pinned. Pipelines that shortlist then decide must report every call: the shortlist call, the rerank call, and any fallback on refusal. Over-cap behavior gets its own cell with its own price: a 400 rejection counts as a failure and its spend still counts, because the caller paid for the attempt.

Token accounting: record raw token classes from billing records, price at dated list rates, and never blend without showing the ratio. For each API contender the runner records input, output, cache-hit, and cache-read tokens as the provider meters them, plus the tier of each call: standard, batch, cached, or off-peak. Spend is computed at list rates in force on the published run date, with the provider invoice total beside it as a cross-check. Ratios may be published for readability, following Artificial Analysis with its stated 7:2:1 blend, but the raw class counts must ship in the machine-readable cells so readers can reprice under any ratio. Retries, refusals, and malformed outputs count in full: tokens spent on failed attempts are spend, and the nibzard convention of marking retried cells as lower bounds is the minimum disclosure, not the target. For self-hosted and local contenders the runner records wall time, batch size, and concurrency instead, and converts to dollars through the utilization rules below. TypeSafe’s pricing of $0.042 per million input tokens with free output tokens that are too cheap to meter shows why the classes stay separate: a contender with free output is nearly immune to answer-length growth and fully exposed to option-count growth, and a blended figure hides both facts.

Error budgets: fix 1 percent and 5 percent empirical budgets and make coverage at budget the denominator. The headline ratio divides spend by accepted decisions, not attempted ones. The benchmark fixes two empirical error budgets, 1 percent and 5 percent, and defines coverage as maximum threshold-realizable coverage at each budget, the AbdelStark definition. Every contender reports its coverage curve, the threshold realizing each budget point, and Brier score, NLL, and ECE beside it. Two verified papers support this shape: best performance on a predetermined coverage ratio with selection from the classifier’s own scores (arXiv:2206.09034), and selective calibration, well-calibrated uncertainty over accepted examples (arXiv:2208.12084). Both budgets are required because one hides the tradeoff: AbdelStark’s pilot shows Jev covering 0.860 of Banking77 at 5 percent error while covering 0.000 of DAIR Emotion at the same budget. A forced-uncertainty slice is mandatory, following the nibzard suite where LLMs admit ignorance on 97.3 to 100 percent of items and Jev on 49.7 percent, because a benchmark without unanswerable items cannot distinguish calibrated abstention from bluffing. Bootstrap intervals ship with every coverage figure. To see why the denominator matters, take the nibzard measured $70 per million for Jev on banking-style prompts and divide by AbdelStark’s 0.860 banking coverage: $70 divided by 0.860 is about $81.40 per million accepted decisions (computed). On a condition with 0.020 coverage the same arithmetic multiplies spend by 50.

Serving utilization: fix three load scenarios and require a utilization sweep with cold starts stated. This factor is the largest source of non-comparability in self-hosted figures, so it gets the strictest rules, borrowed from MLPerf. The three required scenarios are single-stream latency (one outstanding request; report p50 and p95), Poisson server (stated arrival rate with a latency bound; report maximum sustainable rate and spend per decision), and offline throughput (all items at once; report decisions per second and spend per decision at saturation). Each scenario runs 600 seconds by default, matching MLPerf, so results compare against the wider serving literature. The runner then sweeps utilization, reporting dollars per decision at 10, 50, and 100 percent of the maximum sustainable Poisson rate on the named card at the dated list rate. Billing mode is part of the label, since per-second billing with scale to zero and reserved hourly instances produce different curves by construction. Cold starts are measured and stated separately: Laya’s README reports median checkpoint reloads of 7.4 s on CPU and 10.3 s on T4, which bound how cold a scale-to-zero pool can run. Batching delay is reported as added latency at each batch size, not hidden inside throughput: Laya’s GB10 decomposition of roughly 93 ms fixed overhead plus 7.0 ms per extra question is the model for this disclosure. Each API figure names its tier. Each local figure names the machine, the threading, and the batch shape, and power cost stays UNVERIFIED unless metered at the wall.

Reporting rule: the headline figure is a labeled tuple, never a bare number. A comparable dollars-per-million figure carries five labels: suite and item hash, error budget, load scenario, utilization or tier, and price date. The machine-readable cells carry the raw counts behind each label: tokens by class, attempts by outcome, latency percentiles, batch sizes, and the coverage curve. List-price arithmetic and measured spend stay in separate, cross-checked columns.

Implications for small decision models

For Jev, the design clarifies both its strength and the measurement it still owes. TypeSafe’s launch post reports $0.042 per million input tokens with free output, 70 to 500 ms end-to-end latency, 40x-200x faster for System One shaped queries, and homepage figures of 193.6 times faster and 444.6 times cheaper which it expects to sit at the higher end of real-world gains. Jev’s flat latency curve simplifies the server scenario, but three design features bite. First, option-count stratification shows input-token growth on high-cardinality suites with no output-side offset. Second, the sharp 255 cap forces an over-cap cell where every LLM row continues to 512. Third, the error budget divides by coverage, and the two public calibration readings differ: ECE 0.246 with 49.7 percent ignorance admission in one benchmark, and 0.860 banking coverage at 5 percent error with 0.000 emotion coverage in the other. The full behavior of Jev is surveyed in Jev and alternatives; this benchmark would convert its open calibration questions into priced coverage curves.

For Laya, the design rewards what the project already publishes and prices what it discloses as weak. Laya’s published batch curve gives the utilization sweep a favorable shape at saturation. The weak corners get priced rather than hidden: Banking77 at 0.425 against Jev’s 0.870 shows in the high-cardinality stratum, the option-token budget binds as option text grows, and coverage at budget reflects the shipped temperature clamp and the entropy-confidence gating the project documents. The Laya teardown in this graph records the action-head saturation and the clamp in detail; under this benchmark those findings become coverage deltas at fixed budgets rather than footnotes.

For Maestro Flash, which the proposal describes as a design with no weights and no model results, the benchmark sets four constraints before any training run. First, decoupled option encoding must be tested on the stratified cardinality ladder, because the ladder distinguishes a fix for joint-encoding limits from a low-cardinality demo. Second, per question-type, option-count, and language temperatures must ship as coverage curves at both budgets, not as single ECE values, because the accepted-decision denominator decides automation spend. Third, time per decision and the full utilization sweep must accompany every dollars-per-million claim from the first stage-0 harness run; the proposal’s Day 1 to 7 exit condition, approximately reproducing Laya’s published figures on Laya’s own checkpoints, is this benchmark’s reproducibility requirement applied to the project. Fourth, the over-cap cell must be designed now, not after the architecture freezes, since shortlist-then-decide, decoupled encoders, and hierarchical decisions have different cost curves past any single-pass limit.

Limitations

List prices move, and this report captures a single day: promo windows, peak schedules, and batch discounts can halve API rows within a quarter. Measured spend is suite-specific: long 77-way banking prompts overstate short-prompt workloads and understate agentic ones. Self-hosted math omits engineering time, orchestration, monitoring, redundancy, and multi-replica overhead. Local power and hardware amortization are UNVERIFIED without wall metering and a stated depreciation schedule. Outside the error budget, quality stays unequal: the accepted-decision figure compares automation value, not capability. The public pilots are small: AbdelStark’s 300 items give wide intervals on hard slices, and the nibzard v1 to v2 corrections show that protocol bugs happen even in careful harnesses. Routing evidence transfers weakly: FrugalGPT’s 98 percent and RouteLLM’s 2 times figures concern LLM question answering, not typed decisions, and cascade behavior on calibrated abstention is UNVERIFIED. MLPerf scenarios were built for datacenter systems and need adaptation for scale-to-zero serverless and local CPUs. Tokenizer differences across APIs mean input-token counts for identical prompts will disagree; the benchmark records provider-metered tokens and does not attempt a universal tokenizer.

Open questions

1. What reference prompt suite, with fixed option-count strata, state-length bands, pinned revisions, and item hashes, should the benchmark freeze so that tokens per decision stay stable enough to compare a new contender against published dollars-per-million figures without rerunning every baseline?
2. How should the benchmark price the over-cap region past single-pass and vendor choice limits so that shortlist-then-decide pipelines, decoupled option encoders, and hierarchical decisions report one comparable accepted-decision figure?
3. What load-scenario trio and utilization sweep should the benchmark require so that rented-GPU dollars per decision at stated latency bounds compares fairly against API dollars per decision at stated tiers, including cold starts and batching delay?

References

1. FrugalGPT, arXiv:2305.05176: https://arxiv.org/abs/2305.05176
2. RouteLLM, arXiv:2406.18665: https://arxiv.org/abs/2406.18665
3. PagedAttention and vLLM, arXiv:2309.06180: https://arxiv.org/abs/2309.06180
4. HELM, arXiv:2211.09110: https://arxiv.org/abs/2211.09110
5. Calibrated Selective Classification, arXiv:2208.12084: https://arxiv.org/abs/2208.12084
6. Towards Better Selective Classification, arXiv:2206.09034: https://arxiv.org/abs/2206.09034
7. GLiClass, arXiv:2508.07662: https://arxiv.org/abs/2508.07662
8. nibzard/decision-model-benchmark: https://github.com/nibzard/decision-model-benchmark
9. AbdelStark/jev-benchmarks: https://github.com/AbdelStark/jev-benchmarks
10. NandhaKishorM/laya: https://github.com/NandhaKishorM/laya
11. Laya BENCHMARKS.md: https://raw.githubusercontent.com/NandhaKishorM/laya/main/BENCHMARKS.md
12. TypeSafe, Introducing System One Models and Jev: https://typesafe.ai/blog/introducing-system-one-models-and-jev
13. vLLM project README: https://github.com/vllm-project/vllm
14. knowledgator/GLiClass README: https://github.com/knowledgator/GLiClass
15. MLPerf inference rules: https://raw.githubusercontent.com/mlcommons/inference_policies/master/inference_rules.adoc
16. Artificial Analysis benchmarking methodology: https://artificialanalysis.ai/methodology
17. DeepSeek API pricing: https://api-docs.deepseek.com/quick_start/pricing
18. Anthropic pricing: https://www.anthropic.com/pricing
19. Together AI pricing: https://www.together.ai/pricing
20. Modal pricing: https://modal.com/pricing
21. Lambda GPU cloud pricing: https://lambda.ai/pricing

← All papers