Self-Host Crossover Throughput: When a Rented Encoder Beats the Cheapest API

The question

This report answers one question. At what sustained decisions per second does a Maestro-class open encoder on a current budget graphics processing unit (GPU) undercut the cheapest measured application programming interface (API) on identical prompts, and how do cold starts and batching delays move that crossover? Sustained decisions per second (dps) means the long run average arrival rate, not a burst peak, measured under a stated latency bound in the style of the MLPerf server scenario. Identical prompts means the same option lists and state passages on both sides, so tokens per decision and accuracy are held fixed while only the serving path changes.

Three definitions fix scope. A Maestro-class encoder means a 300 to 450 million parameter bidirectional encoder serving typed decisions in one pass, the shape Laya’s README reports at 421M for ModernBERT-large English and 322M for mmBERT-base multilingual. A budget GPU means a low cost cloud card such as T4, L4, A5000, RTX 3090, or A10. The cheapest measured API means Jev at $0.07 per 1,000 on banking, which the nibzard benchmark reports with a median (p50) latency of 264 to 276 ms. The next cheapest is gpt-5.4-nano at $0.19 per 1,000.

The parent paper decision economics already publishes full list price and measured spend tables, and the follow up cost per accepted decision benchmark already designs the accepted decision denominator. This report does not repeat those tables. It adds the crossover layer: the rate at which rented time beats metered tokens, plus the two mechanisms that move that rate in practice, cold starts that tax idle gaps and batching delays that trade latency for unit cost.

Every number below comes from a source fetched for this report on 2026-09-23 and is attributed inline. Computed figures show their arithmetic. Anything not verified is marked UNVERIFIED. Nothing here claims Velofy has trained or shipped any model.

Background

Three billing modes produce different crossover curves by construction. Reserved hourly instances charge for wall clock, so Lambda lists 1x RTX 6000 at $0.69, A10 at $1.29, A100 40GB at $1.99, and H100 SXM at $4.29 per GPU hour. Per-second serverless charges for busy time plus configured warm idle, so Modal lists T4 at $0.000164, L4 at $0.000222, and A10 at $0.000306 per second, and states that the buyer never pays for idle resources, only actual compute time. RunPod splits the same way, with serverless workers that its overview describes as pay for actual compute time with no idle costs, listing 24GB workers at $0.69 per hour against L4 pods at $0.49 and RTX A5000 pods at $0.27.

Throughput is only defined against an arrival process plus a latency bound. The MLPerf inference rules define Single stream (next query on completion), Server (Poisson arrivals, metric is maximum supported Poisson throughput), Offline (all samples at once, metric is measured throughput), and Multistream. The fetched rules run each scenario for 600 seconds, require at least 24,576 samples for the offline query, and apply benchmark specific latency constraints. A dollars per decision figure without arrival shape and latency bound is undefined for every self hosted contender.

Cold starts share one anatomy across vendors. Modal’s cold start guide names two sources: inputs queueing for a warm container, and first invocation initialization such as loading weights from disk to memory. RunPod’s serverless overview defines a cold start as the interval from a request with no running workers to a warmed up worker, covering container start, model load into GPU memory, and runtime init, with larger models slower to load. Laya’s README gives the encoder instance: median checkpoint reloads of 7.4 s on the central processing unit (CPU) and 10.3 s on T4, load improved from about 22 s to about 2 s on CPU, and script detection under 0.5 ms, so model load dominates startup by orders of magnitude.

Batching is the throughput lever. The vLLM README lists continuous batching, chunked prefill, and prefix caching on PagedAttention, and the PagedAttention paper (arXiv:2309.06180) reports 2 to 4 times throughput at the same latency against FasterTransformer and Orca. Encoders get a simpler version: the GLiClass README reports approximately 10 times faster than cross encoder models via a single forward pass, and the GLiNER abstract (arXiv:2311.08526) frames its encoder as parallel extraction against slow sequential token generation. Laya’s README makes the curve concrete on a T4: 1 question at 39.5 ms English and 32.8 ms multilingual, 10 at 158.6 (15.9 per question) and 72.3 (7.2 per question), and 50 at 771 and 337 ms (6.8 per question multilingual), or 103 to 332 questions per second (q/s) on one T4.

Two routing results and two encoder results anchor costs. FrugalGPT (arXiv:2305.05176) reports large language model (LLM) API fees differing by two orders of magnitude, with a cascade matching the best LLM at up to 98 percent lower cost or beating GPT-4 by 4 percent at the same cost. RouteLLM (arXiv:2406.18665) reports over 2 times cost reduction in certain cases without quality loss. ModernBERT (arXiv:2412.13663) reports training on 2 trillion tokens with native 8192 length, calling ModernBERT the most speed and memory efficient encoder, designed for common GPUs. DeBERTaV3 (arXiv:2111.09543) reports 91.37 percent on the General Language Understanding Evaluation (GLUE) benchmark for its Large model. Routing compresses API spend while efficient encoders raise the self hosted ceiling.

What the evidence says

The API side has one clear measured floor. The nibzard benchmark reports Jev at $0.07 per 1,000 on banking (S1, 77-way) with p50 264 to 276 ms flat from 2 to 255 options, calling Jev cheapest by ten times against $0.19 for gpt-5.4-nano. Its S1 column runs from $0.07 to $2.48 across nine contenders, with $28.34 total measured spend over five suites and 60 cells. Retried cost cells are lower bounds marked with a dagger, and items rebuild byte identical from a pinned seed. TypeSafe’s launch post reports the list price behind the meter: $0.042 per million input tokens with output free as too cheap to meter, against frontier input of $0.20 to $10 with output about 5 times input, plus 70 to 500 ms latency and 40x-200x faster for System One shaped queries.

The encoder side has one measured throughput curve and one independent CPU anchor. Laya’s BENCHMARKS.md adds that every checkpoint answered byte identical questions per run and that its Jev figures are third party published, never measured there. Laya’s README adds preloaded serving at 32.8 ms on GPU and 193 to 464 ms on CPU with no reloads, and the 7.4 s CPU and 10.3 s T4 reload medians on eviction. AbdelStark reports GLiNER2.5 near 44 ms p50 on an Apple M4 Max CPU against hosted Jev at 236 to 256 ms from France on small label tasks, with Jev ahead 246 ms versus 296 ms on the 72 label condition. Hugging Face records show ModernBERT-base at 4,372,504 downloads and gliner2.5-multi-v1 at 243,120, which measures adoption, not quality.

The rental side is fully priced for budget cards. Modal adds CPU at $0.0000131 per core per second and a $30 per month starter allowance. RunPod’s page, updated September 13 2026, lists A5000 at $0.27, L4 at $0.49, RTX 3090 at $0.50, and RTX 4090 at $0.74 per hour for pods. Lambda adds 8x H100 SXM at $3.99 per GPU hour. No vendor here publishes encoder decisions per second, so every throughput cell below is either Laya’s measured T4 curve or UNVERIFIED.

The table computes crossover against $70 per million (computed as $0.07 per 1,000 times 1,000). Crossover means the sustained rate above which GPU rental alone is cheaper per decision than the API meter, before engineering, redundancy, and quality differences.

Two checks bound the table. Dividing $70 per million by TypeSafe’s $0.042 list price implies about 1,667 input tokens per decision (computed), a long 77 option prompt, so short prompts cross sooner. AbdelStark reports Jev coverage of 0.860 on Banking77 at 5 percent or lower error, so $70 divided by 0.860 is about $81.40 per million accepted decisions (computed), moving every crossover down about 14 percent on the accepted denominator.

Analysis

The crossover formula is one line: r* equals G divided by (3600 times A), where G is GPU dollars per hour and A is API dollars per decision. With A at $0.00007 ($70 per million), the denominator is $0.252 (computed), so r* is G divided by 0.252: $0.5904 gives 2.34, $0.49 gives 1.94, $1.29 gives 5.12. Against gpt-5.4-nano at $190 per million ($0.00019), the denominator is $0.684 (computed) and the Modal T4 crossover falls to 0.86 dps (computed). Jev is the hardest API to beat; every other row crosses sooner.

Reserved billing makes the crossover a utilization tax. A Modal T4 costs $0.49 to $1.59 per million at full 103 to 332 q/s utilization (computed), 44 to 143 times cheaper than $70 per million, but about $16.40 per million at 10 dps and about $164 per million at 1 dps (computed) because idle time is billed. The 2.3 dps crossover sits at about 1 to 2 percent of the measured ceiling: vastly cheaper at load, more expensive than the API whenever mostly idle. RunPod pods and Lambda instances share this shape with no scale to zero.

Serverless flips the low end but adds colds. At 7.2 ms per question, a Modal T4 at $0.000164 per second prices one decision at about $1.18 per million before colds (computed), which beats $70 at any rate if colds were free. A 10.3 s T4 reload costs about $0.00169 (computed), or $1,690 per million over one decision. Modal defaults to 60 seconds maximum idle with scaledown_window from two seconds to twenty minutes and bills idle GPU reservation; RunPod recommends cached models, FlashBoot, or active workers above zero. One decision per minute with gaps over 60 seconds pays about one cold per decision and loses badly, while 60 decisions in one minute per hour pays one cold per 60, about $28 per million cold amortization plus $1.18 service (computed), and still beats the API. Sustained dps without the gap histogram cannot price serverless.

Keep warm converts serverless back into reserved at a chosen floor. Modal documents min_containers as a floor against scale to zero and buffer_containers for bursty patterns. Setting min_containers to 1 on a Modal T4 means paying $0.5904 per hour continuously, the reserved math above, to remove queueing colds. Keep warm wins when expected cold cost per hour exceeds $0.5904, which at $0.00169 per cold is about 349 colds per hour (computed), or one every 10 seconds. Above that frequency the pool is effectively always warm and should be priced as reserved. Laya’s guidance agrees: preload every served checkpoint so flips cost detection only.

Batching moves the crossover along the latency axis. Laya’s multilingual T4 curve shows 1 question in 32.8 ms, 10 in 72.3 ms (7.2 per question), and 50 in 337 ms (6.8 per question). Batch 10 adds about 39.5 ms to first question latency (computed) while cutting service time about 4.6 times (computed); batch 50 adds about 304 ms (computed) for only 6 percent further gain. At 2 dps, gathering 10 takes 5 seconds of waiting, far above Jev’s measured 264 to 276 ms p50, so fixed large batches are unreachable at crossover rates under API parity latency. The fix is timeout or continuous batching of concurrent arrivals, the vLLM pattern. Under a 300 ms p50 bound, batch 10 at 72.3 ms fits while batch 50 at 337 ms does not, so the crossover should be quoted at the capped throughput near 10, not the 332 q/s offline ceiling.

Three multipliers shift every number above. First, option count is a cost dial on token APIs and an accuracy dial on joint encoders: Laya’s README documents head budgets of 192 and 256 tokens with about 3 to 4 tokens per Banking77 label, reporting 0.425 against Jev’s 0.870. Identical 77 option prompts price different goods if accuracy differs, hence the $81.40 accepted decision correction. Second, Jev’s hard 255 cap, which nibzard reports as 400 Too many choices at 256 and above with 100 percent at 254 and 255, forces larger decisions through shortlist pipelines that roughly double calls, while LLM rows continue to 512. Third, no Jev batch tier was fetched for this report, so Jev batch pricing is UNVERIFIED and $70 stands as standard tier measured spend.

Implications for small decision models

For Jev, surveyed in Jev and alternatives, the crossover confirms strength and exposure. TypeSafe reports parallel single query outputs with $0.042 input and free output, and nibzard measures flat 264 to 276 ms p50 from 2 to 255 options at $0.07 per 1,000. Flat latency removes API-side batch tuning, but $70 per million is still 44 to 143 times the computed full utilization T4 cost, so past low single digit dps on identical long prompts the incentive is to self host or seek volume pricing. The 255 cap, 13 percent flip rate, expected calibration error (ECE) of 0.246, and 49.7 percent ignorance admission, all reported by nibzard, force shortlisting and review spend outside any per token table.

For Laya, torn down in the Laya teardown, the crossover is the deployment guide. It already publishes the batch curve (103 to 332 q/s T4), the cold costs (7.4 s CPU and 10.3 s T4 reloads), and the warm state (32.8 ms GPU, 193 to 464 ms CPU preloaded). The operational reading: preload every served checkpoint, keep two resident when traffic mixes languages, set the idle window from the gap histogram rather than the 60 second default, and cap batches near 10 under a 300 ms p50 bound since batch 50 at 337 ms exceeds it. Near chance zero shot baselines, the option token budget, and per bucket calibration (BENCHMARKS.md reports ECE 0.466 and 0.314 as shipped, 0.081 and 0.106 after refit) are crossover risks priced per domain.

For Maestro Flash, which the proposal and technical report describe as a design with no weights and no model results, the crossover sets four constraints before training. First, decouple option encoding from state context as proposed, since per option tokens dominate API spend and the joint budget binds self hosted accuracy at 77 options. Second, preserve the single pass batch shape and publish batch 1, 10, and 50 latencies on the named card from the first stage zero run. Third, design the warm pool now: preload by default, document cold reload seconds, and report serverless figures at stated scaledown windows. Fourth, ship per question type, option count, and language temperatures as coverage curves at fixed error budgets. The stage zero exit condition of reproducing Laya’s T4 figures on Laya’s checkpoints tests the harness before any new model.

Limitations

List prices move: Modal, RunPod’s September 13 2026 page, and Lambda rows are one day’s capture, and promos or peak schedules can halve API rows within a quarter. Measured spend is suite specific: nibzard’s long 77-way banking prompts overstate short workloads and understate agentic ones. Self hosted math omits engineering, orchestration, monitoring, redundancy, and egress. Power and amortization are UNVERIFIED. Throughput on every card except Laya’s T4 is UNVERIFIED, and serverless cold seconds for a 322 to 421M encoder on a budget worker are UNVERIFIED end to end. Quality stays unequal outside the error budget, so dollars per million is not dollars per accepted decision except where coverage is divided in explicitly. Tokenizers differ across APIs; this report uses provider metered spend. Region was not varied: AbdelStark calls Jev from France, and TypeSafe notes West Coast evals near its service.

Open questions

1. What gap histogram and scaledown sweep should a crossover benchmark require so that one sustained dps figure prices reserved, keep warm, and scale to zero pools on the same arrival trace?
2. What timeout batching policy maximizes decisions per dollar under a 300 ms p50 bound for a Maestro-class encoder on a budget GPU, and how should its added gathering delay be reported separately from service time?
3. What identical prompt suite, with fixed option count strata and state length bands plus pinned revisions and item hashes, should freeze tokens per decision so that a new crossover measurement compares against published dollars per million without rerunning every baseline?

References

1. nibzard/decision-model-benchmark
2. TypeSafe, Introducing System One Models and Jev
3. NandhaKishorM/laya
4. Laya BENCHMARKS.md
5. AbdelStark/jev-benchmarks
6. Modal pricing
7. Modal cold start performance guide
8. RunPod GPU pricing
9. RunPod serverless overview
10. Lambda GPU cloud pricing
11. ModernBERT-base on Hugging Face
12. GLiNER 2.5 multi on Hugging Face
13. vLLM project README
14. knowledgator/GLiClass
15. MLPerf inference rules
16. FrugalGPT, arXiv:2305.05176
17. PagedAttention and vLLM, arXiv:2309.06180
18. ModernBERT, arXiv:2412.13663
19. GLiNER, arXiv:2311.08526
20. RouteLLM, arXiv:2406.18665
21. DeBERTaV3, arXiv:2111.09543

← All papers