What a Million Decisions Costs: APIs, Small Encoders, and Local Hardware

The question

This report answers one question: what does a million typed decisions cost when the same work is bought three different ways, as calls to a commercial application programming interface (API), as inference on a rented graphics processing unit (GPU) running a small open encoder, or as inference on local hardware that is already paid for. A typed decision here means one call that takes unstructured state plus a declared schema and returns a label, a score, or a probability, with no generated text. That is the interface described in this graph’s Jev survey and Laya teardown, and this report does not repeat their behavioral findings. It adds the cost layer on top of them.

The question matters because typed decisions are a volume game. A classifier inside a support queue, a fraud filter, or a workflow router can see millions of calls per month, and at that volume unit cost decides the architecture. Getting the arithmetic right requires separating three things often quoted as one number: the vendor’s list price per token, the measured spend per decision on a real prompt, and the loaded cost of serving a model yourself.

Every number below comes from a source fetched for this report on 2026-09-23, and each is attributed inline to the page, file, or abstract that reports it. Where this report combines two fetched numbers into an estimate, the inputs and the arithmetic are shown so the reader can rescale them. Anything not verified is marked UNVERIFIED. Nothing here is a claim that Velofy has trained or shipped any model.

First principles

The arithmetic in this report rests on five simple relations. Each names the quantity that drives the cost, and so the lever that moves it.

On a token-priced API, cost follows prompt length. Cost per decision equals input tokens times the input price, plus output tokens times the output price. A typed decision has a short answer, so input dominates. Every option label must appear in the request, so cost grows with option count.

On a rented GPU, cost follows utilization. Cost per decision equals the hourly rate divided by the decisions actually served in that hour. The card is billed whether or not it is busy. The same card can be very cheap at full load and expensive at low load.

On owned hardware, the marginal cost is near zero and throughput is fixed. The machine is already paid for, so one more decision costs little beyond power. The constraint is the ceiling on decisions per second, not dollars.

Fixed rent against a per-decision price gives a break-even volume. Monthly rent divided by the API’s price per decision is the monthly volume above which renting is cheaper. Below it, the API wins.

Cost only compares like with like when quality is held fixed. A decision whose probability cannot gate automation still needs a human. Dividing cost per decision by the share of decisions that clear a stated error budget gives cost per accepted decision, the figure that decides automation spend.

Background

Three pricing regimes cover nearly all typed-decision deployments. The first is per-token API pricing: the buyer pays for input tokens and output tokens at list rates, sometimes with discounts for cached prefixes or for batch (offline) traffic. The second is per-second or per-hour GPU rental: the buyer pays for the accelerator whether or not it is busy, so the cost per decision depends on throughput and utilization, not just on the hourly rate. The third is sunk local hardware, typically a laptop or workstation with a capable central processing unit (CPU) or integrated GPU: the marginal cost of one more decision is near zero, but the throughput ceiling is fixed and the machine cannot serve anyone else while it is busy.

Per-token prices do not translate directly into per-decision costs, for three reasons. First, prompt length grows with the decision: a 77-option banking intent prompt carries far more input tokens than a 4-option news prompt, because every option label must appear in the request. Second, output pricing differs wildly across vendors: TypeSafe’s launch post reports free output tokens for Jev because they are too cheap to meter, while the same post reports frontier-model output at roughly 5 times input price. Third, measured spend per decision reflects retries, refusals, and the benchmark’s exact prompts, so it is the honest number for that workload and the wrong number for any other. This report therefore keeps list-price arithmetic and measured spend in separate tables and checks them against each other.

The academic starting point for inference cost is FrugalGPT (Chen, Zaharia, and Zou, arXiv:2305.05176), whose abstract reports that popular large language model (LLM) APIs have heterogeneous pricing structures with fees differing by two orders of magnitude, and proposes three cost-reduction strategies: prompt adaptation, LLM approximation, and LLM cascade. Its abstract reports that a learned cascade can match the best individual LLM with up to 98 percent cost reduction, or improve accuracy over GPT-4 by 4 percent at the same cost. A newer routing paper, CASTER (arXiv:2601.19793), reports up to 72.4 percent inference cost reduction against strong-model baselines while matching success rates, and claims to outperform both heuristic routing and FrugalGPT, though its abstract notes the evaluation uses LLM-as-a-judge scoring. The shared lesson is that routing easy decisions to cheap models is the oldest and best-evidenced cost lever, and it predates every system compared here.

What the evidence says

List prices for API inference cluster into three bands: sub-dollar small models, one-to-two dollar midrange models, and frontier models at four dollars and up per million input tokens. Every figure in the first table is a fetched list price, per one million tokens, input and output.

Two discounts recur across vendors. OpenAI, Anthropic, and Google all document batch APIs at 50 percent off for offline-tolerant traffic, and DeepSeek documents off-peak rates at half of peak plus cache-hit input at $0.006 per million against $0.30 cache-miss at peak, a 50 times gap (computed). Any cost model that ignores batching and caching overstates API spend for scheduled workloads.

Measured spend per decision tells a different and more honest story, because it includes real prompt lengths. The independent benchmark by nibzard/decision-model-benchmark reports cost per 1,000 decisions on its banking suite (S1, 77-way) alongside median (p50) latency, with $28.34 in total measured provider spend across five suites and 60 cells. Its v1 grid alone cost $28.14, with a $40 soft cap and a $60 hard cap enforced by the runner. Multiplying its per-1,000 figures by 1,000 gives measured dollars per million decisions:

The benchmark’s own summary is that Jev is cheapest by ten times, that its p50 is 1.2 times faster than gpt-oss-120b on Cerebras and 10 to 16 times faster than thinking-mode models, and that TypeSafe’s 40x-200x claim holds only against LLMs in their slowest default mode. TypeSafe’s launch post reports 70 to 500 ms end-to-end, 40x-200x faster for System One shaped queries, and homepage figures of 193.6 times faster and 444.6 times cheaper which it expects to sit at the higher end of real-world gains. AbdelStark/jev-benchmarks reports Jev at 236 to 256 ms from France against local GLiNER near 44 ms p50 on an Apple M4 Max CPU for 4- and 6-label tasks, with Jev slightly faster on the 72-label condition (246 ms versus 296 ms).

Self-hosted economics start from encoder throughput and GPU rent. Laya’s README reports 33 ms for one question and 7.2 ms per question batched on a T4, with 39.5 ms (English) and 32.8 ms (multilingual) single-question latency, 72.3 ms for 10 batched multilingual questions, 337 ms for 50 (6.8 ms per question), and batched throughput of 103 to 332 questions per second on a single T4. The same README reports CPU latency of 193 to 464 ms, router script detection under 0.5 ms, and checkpoint reload medians of 7.4 s on CPU and 10.3 s on T4. Laya’s BENCHMARKS.md adds that every checkpoint answered byte-identical questions per run and that its Jev figures are third-party published, never measured there. GLiClass’s README reports classification approximately 10 times faster than cross-encoder models via a single forward pass. The ModernBERT abstract reports training on 2 trillion tokens with native 8,192 sequence length and describes ModernBERT as the most speed and memory efficient encoder, designed for inference on common GPUs. The GLiNER abstract frames its bidirectional encoder as parallel entity extraction against the slow sequential token generation of LLMs, and reports beating ChatGPT in zero-shot named entity recognition (NER) evaluations.

GPU list prices, all fetched, anchor the rental side. Modal bills per second: T4 at $0.000164, L4 at $0.000222, A10 at $0.000306, A100 40GB at $0.000583, A100 80GB at $0.000694, H100 at $0.001097, H200 at $0.001261, B200 at $0.001736, and B300 at $0.001972, with CPU at $0.0000131 per core per second and a $30 per month free compute allowance on starter. RunPod lists per-hour pods including RTX A5000 at $0.27, L4 at $0.49, RTX 3090 at $0.50, RTX 4090 at $0.74, L40S at $1.09, A100 at $1.59, H100 PCIe at $2.89, H100 SXM at $3.49, H200 at $4.59, and B200 at $6.79, plus serverless workers at $0.69 per hour for 24GB classes and $0.58 for 16GB classes. Lambda lists single-GPU instances including RTX 6000 at $0.69, A6000 at $1.09, A100 40GB at $1.99, GH200 at $2.29, H100 PCIe at $3.29, H100 SXM at $4.29, and B200 at $6.99, with 8x H100 SXM at $3.99 per GPU per hour. Hugging Face API records show answerdotai/ModernBERT-base at 4,372,504 downloads and fastino/gliner2.5-multi-v1 at 243,120 downloads, which measures adoption and packaging maturity, not quality.

Local hardware contributes one anchor number: AbdelStark’s benchmark reports GLiNER near 44 ms p50 on an Apple M4 Max CPU, about 23 single-stream decisions per second (computed). This graph’s Laya teardown smoke test reports a mean of 79.6 ms per call for three-question batches on Apple Silicon. Neither run meters power, so local electricity cost is UNVERIFIED, though marginal outlay is effectively zero on an owned machine.

Analysis

To compare list prices on equal footing, define a reference decision of 500 input tokens and 50 output tokens, roughly a short state passage plus a moderate option list and a short structured answer. The formula is cost per 1,000 decisions equals 0.5 times the input price plus 0.05 times the output price, with prices in dollars per million tokens. This is an assumption, stated so the reader can rescale it: double the tokens and every figure below doubles. Under it, computed cost per million decisions is about $21 for Jev, $45 for gpt-5-nano, $59 for Qwen3.8 Flash on Together, $70 for Gemini 2.5 Flash-Lite, $105 for gpt-4o-mini, $210 for DeepSeek flash at peak ($105 off-peak), $750 for Haiku 4.5, and $1,500 for Sonnet 5. Batch discounts halve the OpenAI, Anthropic, and Google rows for offline workloads.

Two consistency checks suggest the reference size is close to real banking-style prompts but slightly small. First, nibzard’s measured $70 per million for Jev, divided by Jev’s $0.042 list price with free output, implies about 1,667 input tokens per decision (computed), roughly three reference decisions, which fits a 77-option prompt. Second, the reference formula prices gpt-5.4-nano at $162.50 per million against nibzard’s measured $190, a 17 percent gap in the expected direction (computed). The practical lesson is that option count is a cost dial on every token-priced API: high-cardinality decisions cost a multiple of low-cardinality ones before any model difference enters.

Self-hosted math flips the structure from per-token to per-hour. Laya’s reported 103 to 332 batched questions per second on one T4, combined with Modal’s listed $0.000164 per T4-second ($0.5904 per hour, computed), gives $0.49 to $1.59 per million decisions at full utilization (computed), with $0.82 at the midpoint rate of 200 per second. The same throughput at RunPod’s listed $0.49 per L4-hour would be about $0.68 per million (computed), but Laya’s throughput was measured on a T4, so the L4 figure is illustrative and UNVERIFIED as a system measurement. At low utilization the picture reverses: an average of 10 decisions per second on that Modal T4 costs about $16.40 per million (computed), because the idle seconds are billed too. Serverless GPU with scale-to-zero exists precisely for this gap: Modal’s per-second billing with no idle charge and RunPod’s serverless tiers at $0.58 to $0.69 per hour change the tradeoff, though cold starts such as Laya’s reported 7.4 s CPU and 10.3 s T4 checkpoint reloads bound how cold the pool can run.

Break-even against the cheapest measured API is reachable. A Modal T4 reserved for a 730-hour month costs about $431 (computed). Against nibzard’s measured $70 per million for Jev, that card breaks even near 6.16 million decisions per month (computed), or about 2.3 sustained decisions per second over the same 730 hours (computed). Above that line, the rented encoder is cheaper per decision; below it, the API wins unless batching or caching closes the gap. Training cost amortizes fast at volume: Laya’s README reports fine-tuning runtime of roughly 4 to 5 hours on two T4s over about 30,000 questions, near $5.31 per run on Modal T4 rates (computed). One run spread over a million decisions adds about $5.31 per million; over a hundred million it vanishes.

Several multipliers hide inside the headline numbers. Jev’s hard 255-choice cap, reported by nibzard as a 400 error at 256 and above, forces any larger decision through a shortlist-then-decide pipeline, roughly doubling calls for high-cardinality work. Laya’s fixed option-token budget creates the mirror cost: its README documents the tradeoff between option length and state context, so high-cardinality accuracy must be bought with shortlisting calls or longer contexts. Retries and refusals add a smaller multiplier on every API. The largest honest multiplier is calibration: cost per decision means little if the probabilities cannot gate automation. AbdelStark’s benchmark reports Jev coverage of 0.830 on AG News and 0.860 on Banking77 at a 5 percent error budget, against GLiNER coverage of 0.240 and 0.270, but 0.000 against 0.020 on DAIR Emotion. Jev’s measured $70 per million on banking therefore prices near $81 per million accepted decisions (computed as $70 divided by 0.860). Any comparison that holds accuracy and calibration unequal is pricing different goods.

Cascades compress all of the above. FrugalGPT’s abstract reports matching the best single LLM at up to 98 percent lower cost, and CASTER’s abstract reports up to 72.4 percent lower cost than strong-model baselines with matched success rates. A cascade serving most banking decisions from a $70-per-million tier and escalating a minority to a $2,480 tier would land in the low hundreds per million. The routing accuracy and the judge quality do all the work in such designs, so CASTER’s LLM-as-judge caveat matters: a cost claim graded by a model is weaker than one graded by outcomes.

Implications for small decision models

For Jev, the economics confirm its positioning and its limits. At a measured $70 per million on banking-style prompts, it is the cheapest API in the independent table by a wide margin, and its latency is flat across option counts where LLM latency grows. But it is still roughly 40 to 140 times the computed full-utilization cost of a rented T4 encoder ($0.49 to $1.59 per million), so any customer past a few million decisions per month has a strong incentive to self-host or to demand volume pricing. Jev’s free output tokens and $0.042 input list price make it nearly immune to answer-length growth, yet fully exposed to option-count growth on the input side. Its 255-choice cap and its measured calibration on forced-uncertainty items, both documented in this graph’s Jev survey, are cost facts as well as quality facts: they force shortlisting pipelines and human review queues whose spend never appears in a per-token table.

For Laya, the economics are the point of the project. An open encoder at 103 to 332 batched decisions per second on a single T4 turns rented compute into sub-$2-per-million decisions at load, with a fine-tune cycle near $5 per run and a CPU fallback in the hundreds of milliseconds. The weaknesses named in this graph’s teardown read as cost risks: near-chance zero-shot baselines mean a new domain pays the data bill before the first cheap decision, the option-token budget means high-cardinality tasks pay a shortlisting tax, and the action head saturating at 1.0 means gating must use the documented entropy confidence until the linked issue is resolved.

For Maestro Flash, which this graph’s proposal describes as a design with no weights and no model results, the cost evidence sets four design constraints. First, batch the single forward pass: Laya’s reported 7.2 ms per batched question against 33 ms serial is a 4 to 5 times cost lever available to any encoder that serves questions together. Second, decouple option encoding from the state-context budget, as the proposal plans, because per-option input tokens are the dominant API-side cost driver and the binding truncation constraint on the self-hosted side. Third, fit calibration per question type, option count, and language, because cost per accepted decision at a fixed error budget is the denominator that decides automation spend, and the proposal’s temperature plan targets exactly that. Fourth, publish tokens per decision and utilization alongside every dollars-per-million claim, since this report shows that both can swing the headline by an order of magnitude. The proposal’s 1,344 H100-hour planning allocation prices near $5,363 at Lambda’s listed 8x H100 SXM rate of $3.99 per GPU-hour (computed), which frames the training budget as thousands of dollars against per-million-decision serving costs in the single dollars: the build cost is recouped quickly if the model serves volume, and never if it does not.

The general implication is a measurement rule for the whole graph: report cost per accepted decision at a stated error budget, with the prompt’s token count and the serving utilization attached. List prices rank vendors; measured spend ranks workloads; only the accepted-decision figure ranks systems.

Limitations

List prices move: this report captures promo windows (Gemini’s 2026 pricing), peak and off-peak schedules (DeepSeek), and batch discounts that can halve API rows, so any figure here can stale within a quarter. Measured spend is suite-specific: nibzard’s banking prompts are long and 77-way, so its dollars-per-million overstate short-prompt workloads and understate agentic ones. Self-hosted math assumes sustained batched throughput on one card and omits engineering time, orchestration, monitoring, multi-replica redundancy, and cold starts. Power and hardware amortization for local runs are UNVERIFIED. Quality is held unequal throughout: accuracy, calibration, language coverage, and choice caps differ across every row, so dollars per million decisions is not dollars per correct or per accepted decision except where coverage is divided in explicitly. Finally, Jev’s tokenization of schemas, Laya-on-L4 throughput, and any CASTER-style routing result graded by outcomes rather than by a model judge are all UNVERIFIED in this report’s sources.

Open questions

1. What would a cost-per-accepted-decision benchmark need to fix (prompt shapes, token accounting, error budgets, serving utilization) so that one dollars-per-million figure is comparable across APIs, rented GPUs, and local hardware?
2. At what sustained decisions-per-second does a Maestro-class open encoder on a current budget GPU undercut the cheapest measured API on identical prompts, and how do cold starts and batching delays move that crossover?
3. How should the cost curve past 255 options be measured and minimized: shortlist-then-decide pipelines, decoupled option encoders, or hierarchical decisions, scored on dollars per accepted decision rather than accuracy alone?

References

1. TypeSafe AI, Introducing System One Models and Jev: https://typesafe.ai/blog/introducing-system-one-models-and-jev
2. nibzard/decision-model-benchmark: https://github.com/nibzard/decision-model-benchmark
3. AbdelStark/jev-benchmarks: https://github.com/AbdelStark/jev-benchmarks
4. NandhaKishorM/laya: https://github.com/NandhaKishorM/laya
5. Laya BENCHMARKS.md: https://raw.githubusercontent.com/NandhaKishorM/laya/main/BENCHMARKS.md
6. OpenAI API pricing docs: https://platform.openai.com/docs/pricing
7. Anthropic pricing: https://www.anthropic.com/pricing
8. DeepSeek API pricing: https://api-docs.deepseek.com/quick_start/pricing
9. Google Gemini API pricing: https://ai.google.dev/gemini-api/docs/pricing
10. Together AI pricing: https://www.together.ai/pricing
11. Modal pricing: https://modal.com/pricing
12. RunPod GPU pricing: https://www.runpod.io/gpu-pricing
13. Lambda GPU cloud pricing: https://lambda.ai/pricing
14. FrugalGPT, arXiv:2305.05176: https://arxiv.org/abs/2305.05176
15. CASTER, arXiv:2601.19793: https://arxiv.org/abs/2601.19793
16. ModernBERT, arXiv:2412.13663: https://arxiv.org/abs/2412.13663
17. GLiNER, arXiv:2311.08526: https://arxiv.org/abs/2311.08526
18. knowledgator/GLiClass: https://github.com/knowledgator/GLiClass
19. ModernBERT-base on Hugging Face: https://huggingface.co/answerdotai/ModernBERT-base
20. GLiNER 2.5 multi on Hugging Face: https://huggingface.co/fastino/gliner2.5-multi-v1

← All papers