Stage Zero: A Fixed Six-Axis Harness for Choosing a Decision Backbone

The question

Which backbone wins a fixed stage-zero decision harness that scores Banking77 high-cardinality choice, ordinal error, Brier score, expected calibration error (ECE), median (p50) latency, and option-order flip rate across ModernBERT-large, mmBERT-base, Ettin-encoder-400M, and DeBERTa-v3-large? This paper does not declare a winner, because no such head-to-head measurement exists in the sources surveyed. It does the next most useful thing: it fixes the harness so the question becomes answerable, collects every verified prior that bears on each axis, and states the decision rule that will pick the winner once the numbers exist.

The parent survey in this graph, Which Encoder Backbone Should a Small Decision Model Start From?, recommends ModernBERT-large for English, mmBERT-base for multilingual, and Ettin-encoder-400M plus DeBERTa-v3-large as ablations. That recommendation rests on pretraining reports, General Language Understanding Evaluation (GLUE) and retrieval figures, and ecosystem signals, not on decision-task measurements. This paper takes that shortlist as given and designs the experiment that tests it. Nothing here claims Velofy has trained or shipped any model. Every number below is attributed to the source that reports it.

First principles

A decision model is a backbone (the pretrained encoder, or trunk) plus small heads that turn the trunk’s vectors into choices, scores, and probabilities. Stage zero asks a narrow question about the first part only, and five constraints follow from that.

• To credit a difference to the trunk, hold everything else fixed. Heads, training budget, option shuffling, items, label strings, and hardware must match across backbones. Anything that varies becomes a second explanation for the result.
• Context length bounds how much each option can say. In a single-pass head, the state and every option label share one input sequence. At 77 labels, a 512-position trunk has at most about 6.6 positions per label (512 divided by 77, computed) before the state takes its share. An 8192-position trunk has a far looser limit, about 106 positions per label (8192 divided by 77, computed), unless a production truncation brings the tight one back. High-cardinality accuracy can therefore measure the input budget rather than what the trunk knows.
• Temperature scaling changes confidence, not ranking. Dividing logits by one fitted number leaves the top answer where it was and moves only the probabilities. A refit can move ECE a long way, while accuracy and flip rate do not move under it at all. That is why ECE is reported before and after the refit, and why Brier score, which mixes discrimination and calibration, is reported but not ranked.
• Flip rate measures training as well as pretraining. Shuffling option order during fine-tuning teaches order invariance directly. The shuffling policy has to be fixed across backbones, and a run without it shows how much stability comes from the trunk itself.
• Latency belongs to a whole stack, not to weights. The same weights run at different speeds on different hardware, batch sizes, and runtimes. Latency can rank backbones only when all of them run on one pinned configuration.

Background

Stage zero is the backbone ablation step in the Maestro Flash proposal: before any multi-task training, distillation, or calibration work, each candidate backbone gets the same decision heads, the same training budget, and the same frozen evaluation, and the recorded measurements pick the trunk. The proposal budgets 96 of its 1,344 H100-hours to this step and sets a day 18 decision between mmBERT-base and Ettin-encoder-400M. This paper widens that comparison to the full four-backbone shortlist from the parent survey and pins down what “wins” means.

The six axes are chosen because each one maps to a documented failure in an existing system, not because they are standard leaderboard metrics. Banking77 high-cardinality choice is the stress test for option encoding: Laya’s README-era checkpoints report 0.425 on Banking77 for both English and multilingual trunks, a figure this graph’s Laya teardown attributes to option labels receiving only a few tokens each at 77 labels. Ordinal error covers the score primitive, which the same teardown calls Laya’s weakest primitive, citing a self-reported five-class Stanford Sentiment Treebank (SST-5) figure of 0.372. Brier score and ECE cover probability quality, the axis where both Jev and Laya look worst: one independent benchmark reports Jev at ECE 0.246 with large language models (LLMs) ranging 0.039 to 0.122, and Laya’s BENCHMARKS.md reports as-shipped ECE of 0.466 for its English checkpoint and 0.314 for multilingual. Latency as p50 per decision is the deployment constraint, with Laya’s README reporting 33 ms for one question and 7.2 ms per question batched on a T4, against independently measured Jev p50 of 236 to 276 ms. Option-order flip rate measures position bias: the same independent benchmark reports Jev changing 13 percent of choices under permutation, while Laya’s BENCHMARKS.md reports 0.150 and 0.230 at 20 options for its two checkpoints.

Two facts from the sibling papers shape the design. First, the two public Jev measurements of Banking77-style accuracy disagree in absolute terms (0.870 on a 72-label BTZSC condition in one benchmark, 76.3 percent on 77-way suites in the other), which is expected because they use different item sets, label wordings, and protocols. The lesson is that the harness must pin one dataset revision, one label table, and one protocol before any backbone runs, or the same backbone will produce two incomparable numbers. Second, Laya’s BENCHMARKS.md states that every checkpoint in each run answered byte-identical questions under a fixed seed, and the jev-benchmarks README describes pinned dataset and checkpoint revisions with deterministic sampling under seed 20260917. The stage-zero harness copies that contract: fixed items, fixed seeds, pinned revisions, byte-identical questions per backbone.

Harness specification

The harness holds everything constant except the backbone weights and tokenizer. Each candidate (ModernBERT-large, mmBERT-base, Ettin-encoder-400M, DeBERTa-v3-large) is wrapped in one shared decision-head implementation with an identical parameter budget above the trunk: a small refinement layer and one scorer per option for choice, following the single-pass pattern that Laya’s teardown and the GLiClass paper both describe, plus an ordinal head for score. Tokenizer differences are unavoidable, so the input budget is specified in characters of state plus a fixed per-option token cap, and the truncation behavior is logged per example rather than silently applied. That logging matters because the backbones report different native lengths: ModernBERT’s config sets max position embeddings to 8192, while DeBERTa-v3-large’s config sets 512, and Laya’s README shows production truncations of 512 and 1024 regardless of native length.

Training is a fixed fine-tuning budget per backbone, not training to convergence: the same optimizer, schedule, batch size, epoch count, and option-order shuffling policy. The shuffling policy is fixed because flip rate is a scored axis; if one backbone saw shuffled options and another did not, the comparison would measure the augmentation, not the trunk. A diagnostic run with shuffling disabled shows how much stability comes from augmentation versus pretraining.

Each axis gets one primary metric and a fixed evaluation condition. Banking77 high-cardinality choice uses accuracy on the full 77-label task, scored with the same label strings for every backbone. The dataset itself is the 13,083-example, 77-intent collection released by Casanueva et al., whose abstract describes it as a challenging single-domain intent detection set. Ordinal error uses mean absolute error (MAE) on a fixed ordinal suite (SST-5 style five-level sentiment plus at least one non-sentiment ordinal task, since sentiment alone cannot represent the score primitive). Brier score is the multiclass mean squared error of the full probability vector, and ECE is top-label expected calibration error with 15 equal-width bins, both computed before and after held-out temperature fitting. The temperature protocol follows the recipe whose effectiveness Guo et al. report: a single-parameter scaling fit on held-out data, one temperature per (question type, option-count bucket) following Laya’s bucketing, clamped to [0.5, 5.0] as Laya’s client does. Latency is p50 per single-question decision on one pinned configuration (the proposal’s T4 reference, batch size 1, after warmup, median of at least 100 timed calls), with 95th-percentile (p95) latency and batched throughput reported as diagnostics. Flip rate is the fraction of decisions whose top-1 choice changes under a fixed set of option permutations (at least 5 seeds) at three cardinalities: 5, 20, and 77 options.

The decision rule is lexicographic with vetoes, not a weighted sum, because weights would smuggle in a judgment the lab has not earned. The vetoes come first: any backbone that cannot serve the 77-option condition within the latency budget, or whose post-refit ECE exceeds a pre-registered ceiling, is out regardless of accuracy. Among survivors, the ranking is Banking77 accuracy first, post-refit ECE second, flip rate at 77 options third, ordinal MAE fourth, and p50 latency fifth. Brier score is reported but not ranked, because it mixes discrimination and calibration and the harness already scores those separately. Every comparison ships with paired bootstrap intervals over the same items, following the jev-benchmarks practice of paired target-stratified intervals, so a win means the interval excludes zero. The full item table, label strings, seeds, revisions, and logs are published with the result, matching the reproducibility contract both third-party Jev benchmarks already meet.

What the evidence says

No source surveyed here runs this harness, so there are no head-to-head decision numbers to report. What exists are priors: verified facts about each backbone that constrain where it should be strong or weak on each axis. The table below collects them. Every entry is attributed to the source named in the rightmost column, and download counts come from Hugging Face (HF) application programming interface (API) queries run on 2026-09-23.

Four patterns in this table matter for the harness. First, every backbone row has the same hole: none of the backbone sources report Brier score, ECE, flip rate, or high-cardinality behavior for their own checkpoints. The parent survey notes this gap, and it is the entire justification for stage zero. GLUE figures (DeBERTaV3-large 91.37 percent, Ettin-large 90.8 against ModernBERT-large 90.4, all author-reported) are existence proofs of discriminative competence, not predictions of decision quality.

Second, the only decision-task numbers anywhere in the table belong to Laya’s two trunks, and they set the floors the harness must beat. Laya’s BENCHMARKS.md reports Banking77 0.425 for both checkpoints, refit ECE of 0.081 and 0.106, per-question T4 latency of 39.5 ms and 32.8 ms unbatched, and flip rates of 0.150 and 0.230 on the 20-option intent suite. The Maestro Flash proposal’s day 28 go/no-go, which requires Banking77 above Laya’s 0.425 to 0.492 range, is calibrated against exactly these figures. A stage-zero result in which no backbone clears 0.425 on Banking77 would be informative, not a failure: it would show the head design, not the trunk, is the binding constraint.

Third, context length is the sharpest architectural divide and the one most likely to interact with the 77-option axis. ModernBERT-large reports 8192 native positions with local attention over 128 tokens and global attention every 3 layers, per its config, while DeBERTa-v3-large reports 512 max position embeddings. In a single-pass head where all option strings share one sequence with the state, 77 labels at 512 positions is a visibly tighter budget than at 8192, and Laya’s documented failure mode (labels reduced to 3 to 4 tokens each) was measured on trunks truncated to 512 and 1024. The harness therefore has a genuine hypothesis to test: whether DeBERTaV3’s RTD pretraining advantage survives a context budget that forces aggressive option truncation, and whether ModernBERT’s long context converts into measured high-cardinality accuracy once truncation is actually relaxed.

Fourth, latency priors cut against the largest models but are all measured on different stacks, so they cannot be compared directly. Laya’s README reports 33 ms single-question and 7.2 ms per question batched on a T4, and its BENCHMARKS.md speed table reports 39.5 ms versus 32.8 ms for one question and 771.3 ms versus 337.4 ms for 50 questions across its two checkpoints. Jev’s independent p50 figures sit at 236 to 276 ms from two separate benchmarks, and GLiClass’s README claims classification about 10 times faster than sequential cross-encoder scoring. None of these were measured under one protocol, which is why the harness pins hardware, batch size, warmup, and call count before ranking anyone on latency.

Analysis

The honest pre-registration is a set of per-axis favorites with stated reasons, not a predicted winner. On Banking77 accuracy, ModernBERT-large is the default favorite: it has the longest reported native context in the English pair, the largest adoption footprint (625,057 downloads against 9,660 for Ettin-encoder-400M per the HF API), and a deploy history on exactly this trunk inside Laya. Ettin-encoder-400M is the live underdog because its model card reports small consistent gains over ModernBERT on GLUE and retrieval at both scales, and its abstract’s controlled encoder-versus-decoder result (a 400M encoder outperforming a 1B decoder on Multi-Genre Natural Language Inference, MNLI) is the most on-point architecture evidence available. DeBERTa-v3-large is the high-variance entry: the highest author-reported GLUE figure in the table at 91.37 percent, but a 512 context that may bind at 77 options. mmBERT-base is expected to trail on English Banking77; its reason for being in the harness is the multilingual question, and its English score is the control that prices multilinguality, as the parent survey argues.

On calibration (Brier and ECE), there is no favorite, and any pre-registered favorite would be fabrication. None of the four backbone sources report calibration, and the closest evidence (Laya’s refit ECE of 0.081 and 0.106 from as-shipped 0.466 and 0.314) shows that temperature fitting dominates trunk choice: fitting moves ECE by 0.2 to 0.4 while the post-fit trunk gap is 0.025. A plausible outcome is that all four land within one bootstrap interval after refitting, in which case calibration drops out of the ranking. That would itself be a finding worth publishing, consistent with Guo et al.’s report that single-parameter temperature scaling is surprisingly effective.

On flip rate, the prior favors no trunk either, but it predicts that the augmentation schedule will matter more than pretraining. Laya’s BENCHMARKS.md reports 20-option flip rates of 0.150 and 0.230 against Jev’s independently measured 0.13, and explicitly recommends more aggressive option-order shuffling during training. If disabling shuffling multiplies every trunk’s flip rate by a similar factor, flip rate is an augmentation axis and the ranking on it should be read with that caveat. DeBERTaV3’s disentangled position encoding is sometimes hypothesized to reduce position bias, but no fetched source measures that claim on decisions, so it stays UNVERIFIED until the harness runs.

On latency, the structural expectation is that mmBERT-base is fastest per parameter count (322M per Laya’s README against the 395M to 421M English trunks), that ModernBERT-large and Ettin-encoder-400M are close, and that DeBERTa-v3-large may be slowest per layer. All of that is reasoning from architecture, not measurement: Laya’s BENCHMARKS.md shows the multilingual checkpoint at 337.4 ms versus 771.3 ms for 50 batched questions, a larger gap than parameter counts alone would predict, plus 93 ms of fixed per-call overhead on GB10 that no backbone choice removes. The veto design reflects this uncertainty: latency eliminates trunks that miss the budget but does not otherwise rank them.

The most likely way this harness surprises its designers is a cross-axis tradeoff with no Condorcet winner: Ettin ahead on accuracy, DeBERTaV3 ahead on flip rate, mmBERT ahead on latency, all tied on calibration. The lexicographic rule resolves that case by construction (accuracy first), but the paper reporting the result should publish the full six-axis table with intervals, not just the ranking, so a reader with a different deployment budget can re-rank. The second most likely surprise is a head-design ceiling: if all four trunks cluster near Laya’s 0.425 on Banking77, the binding constraint is the single-pass option encoding, and the correct follow-up is the decoupled option encoding the Maestro Flash proposal already sketches, not a different trunk.

Implications for small decision models like Jev, Laya, and Velofy’s proposed Maestro Flash

For Jev, this harness changes nothing about what is known and everything about what is comparable. Jev’s architecture remains undisclosed, as this graph’s Jev survey documents, so no backbone result transfers to it. But the harness’s six axes are deliberately the same six on which third-party Jev measurements exist: Banking77-class accuracy (0.870 on 72-label BTZSC in one benchmark, 76.3 percent on 77-way in the other), ECE 0.246, p50 264 to 276 ms, 13 percent flip rate, and a hard 255-choice cap. Any backbone that clears the harness therefore arrives with a behavioral comparison table against Jev ready to fill in, provided the comparison uses the same items and label strings. Cross-system comparisons should stay at or below 77 options, where every system actually answers.

For Laya, the harness targets its two weakest reported numbers: Banking77 at 0.425 on both trunks and 20-option flip rates of 0.150 and 0.230. A stage-zero run varying only the trunk on Laya-style heads would show whether either number moves with the backbone or is pinned by the head design. If the sweep moves nothing, Laya’s documented head-level fixes (more aggressive shuffling, shortlisting, longer head budgets) are confirmed as the right next work. If Ettin or DeBERTaV3 moves Banking77 substantially on identical heads, Laya gains a trunk upgrade with no architecture change.

For Velofy’s proposed Maestro Flash, this harness is the day 18 decision procedure, widened from two backbones to four. The proposal nominates mmBERT-base as the production candidate with Ettin-encoder-400M as the ablation; the parent survey amends that with an English ModernBERT-large control; this paper adds DeBERTa-v3-large as the fourth cell and specifies the vetoes, ranking, and intervals that turn measurements into a decision. The proposal’s downstream gates stay as written: the day 28 Banking77 go/no-go against Laya’s 0.425 to 0.492 range, and the day 63 criterion of lower post-refit ECE at matched accuracy. One sequencing point matters: stage zero should run before the Indic benchmark exists, in English only, because its job is trunk selection and adding languages before selecting the trunk confounds the two decisions. The multilingual cost question (mmBERT against an English control behind a router, per language) is a separate experiment that runs after the trunk is fixed. No training has been done for Maestro Flash, and this paper predicts no winner.

Limitations

This is a measurement design, not a measurement. No backbone was fine-tuned, timed, or scored for this paper, no Velofy checkpoint exists, and every per-axis favorite above is a prior to be tested, not a result. The harness could also be wrong in its fixed choices: 15 ECE bins, 5 permutation seeds, three cardinalities, the specific ordinal suite, and the lexicographic order are all defensible but not derived, and a different lab could fix different values and get a different winner. The Decision Bench v0 freeze, which builds on this harness, settles one of these differently: it uses 10-bin ECE to stay comparable with published decision-model ECE values. A stage-zero run that prints 10-bin ECE beside its 15-bin figure can be read against both. The specification is published so that disagreement can take the form of a rerun rather than an argument.

The evidence base has the same vendor-source weakness the parent survey notes. Ettin’s GLUE and retrieval deltas come from its own model card, DeBERTaV3’s 91.37 percent from its own abstract, mmBERT’s 1800-language breadth from its own abstract, Laya’s floors from its own BENCHMARKS.md, and Jev’s figures from two young third-party benchmarks with different protocols. The harness design treats all of these as priors with wide intervals, which is why the decision rule requires paired bootstrap intervals on shared items before declaring any win.

Three quantities the harness needs are UNVERIFIED at the time of writing: mmBERT-base’s native context length (no fetched source states it), the exact per-option token budget at which each head design saturates (Laya’s 3 to 4 tokens per label at 77 options is reported for its head budgets, not as a general law), and p50 latency for any of these trunks under the pinned harness configuration (every cited latency figure was measured on a different stack). The harness run itself resolves all three by logging truncation per example and timing the pinned configuration directly.

Finally, the harness is English-only and single-domain by design, and its winner is therefore the best English trunk under this head design, not the best backbone in general. Multilingual quality, retrieval use, generation use, and robustness to adversarial or out-of-distribution inputs are out of scope. A trunk that wins stage zero can still lose the multilingual decision, the calibration-at-scale decision, or the production cost decision, and the paper reporting the result should say which decision was actually made.

Open questions

1. What are the measured six-axis results when the fixed stage-zero harness is actually run, with paired bootstrap intervals, across ModernBERT-large, mmBERT-base, Ettin-encoder-400M, and DeBERTa-v3-large under one head implementation and one training budget?
2. How much of each trunk’s option-order flip rate is removed by the shuffling schedule alone, measured by the disabled-shuffling diagnostic run, and does any backbone show a pretraining-level stability advantage that augmentation cannot explain?
3. At what option cardinality does each trunk’s accuracy break under the single-pass head, and does the break point track native context length (8192 for ModernBERT-large, 512 for DeBERTa-v3-large, UNVERIFIED for the others) or a head-design ceiling shared by all four?

References

1. ModernBERT paper, arXiv:2412.13663
2. mmBERT paper, arXiv:2509.06888
3. Ettin paired encoders and decoders paper, arXiv:2507.11412
4. DeBERTaV3 paper, arXiv:2111.09543
5. On Calibration of Modern Neural Networks, arXiv:1706.04599
6. Efficient Intent Detection with Dual Sentence Encoders (Banking77 dataset), arXiv:2003.04807
7. GLiClass paper, arXiv:2508.07662
8. GLiNER paper, arXiv:2311.08526
9. ModernBERT release blog, https://huggingface.co/blog/modernbert
10. ModernBERT GitHub repository, https://github.com/AnswerDotAI/ModernBERT
11. DeBERTa GitHub repository, https://github.com/microsoft/DeBERTa
12. Laya GitHub repository, https://github.com/NandhaKishorM/laya
13. Laya benchmarks, https://github.com/NandhaKishorM/laya/blob/main/BENCHMARKS.md
14. Ettin 400M encoder model card, https://huggingface.co/jhu-clsp/ettin-encoder-400m
15. ModernBERT-large config, https://huggingface.co/answerdotai/ModernBERT-large/blob/main/config.json
16. DeBERTa-v3-large config, https://huggingface.co/microsoft/deberta-v3-large/blob/main/config.json
17. jev-benchmarks, https://github.com/AbdelStark/jev-benchmarks
18. decision-model-benchmark, https://github.com/nibzard/decision-model-benchmark

← All papers