Which Encoder Backbone Should a Small Decision Model Start From?

The question

A small decision model takes unstructured state plus a declared schema and returns typed probabilities in one forward pass, with no generated text. The backbone question is which pretrained encoder should sit underneath the decision heads at the start of such a project, and what evidence supports that choice. This survey answers for the open-weights regime only, at roughly the scale Laya and the Maestro Flash proposal occupy. Laya’s README reports 421M parameters for its English checkpoint and 322M for its multilingual checkpoint, and the Maestro Flash proposal in this graph plans a 322M to 400M model, so the relevant band is on the order of one to four hundred million parameters.

The short answer is that there is no backbone that wins on every axis, so the starting pick should follow the language scope of the product. For English-only decisions, start from ModernBERT-large and ablate against Ettin-encoder-400M. For multilingual decisions, start from mmBERT-base and ablate against an English control plus continued pretraining. For zero-shot classification driven by natural language inference (NLI), consider DeBERTa-v3-large as a second ablation because of its mature NLI classifier ecosystem. Retrieval-tuned encoders such as BGE-M3 and Nomic’s ModernBERT embed are strong retrievers but are the wrong default for a decision backbone, for reasons given below. Nothing in this survey is a claim that Velofy has trained or shipped any model. All figures are attributed to the source that reports them.

Background

A typed-decision model differs from a chatbot and from a search encoder in what it asks of its backbone. The backbone must encode state text and option text into representations from which small heads can score every declared option at once. The Laya teardown in this graph describes the concrete pattern: option markers inside one sequence, a small Transformer refinement layer, then one logit per option. GLiClass uses the same one-pass idea for classification, and its README reports classification results approximately 10 times faster than cross-encoder models that must process text-label pairs sequentially. The backbone therefore needs strong bidirectional representations of short label-like strings as well as longer state passages, in the same forward pass.

Calibration is a second requirement, and it is mostly orthogonal to the backbone. Laya fits one temperature per question type and option count bucket, with values clamped to a fixed interval, as described in this graph’s Laya teardown. The backbone choice cannot substitute for that step, because none of the backbone sources surveyed here report calibration metrics such as expected calibration error (ECE) or Brier score for their pretrained checkpoints. A backbone can only supply the starting discrimination and the language coverage. The decision project must still measure accuracy, calibration, and latency itself.

Latency and context shape the choice further. Laya’s README reports 33 ms for one question and 7.2 ms per question when batched, measured on an NVIDIA T4 graphics processing unit (GPU), which sets the practical envelope: tens of milliseconds per decision on modest accelerators. Context is tighter than marketing suggests. ModernBERT’s abstract reports a native 8,192 token sequence length, and its config file sets max position embeddings to 8,192 with local attention over 128 tokens and global attention every 3 layers, but Laya truncates to 512 or 1024 tokens depending on checkpoint, as its README reports. Option text, instructions, and state compete for that truncated budget, and Laya’s Banking77 weakness is attributed in this graph’s teardown to option labels receiving only a few tokens each at 77 labels. A backbone with long native context helps only if the decision architecture actually uses it.

The last background point is the encoder versus decoder question, which now has a controlled answer. The Ettin paper abstract reports paired encoder-only and decoder-only models from 17M to 1B parameters trained on up to 2T tokens with the same recipe, and finds that encoders excel at classification and retrieval while decoders excel at generation. Its abstract states that a 400M encoder outperforms a 1B decoder on Multi-Genre Natural Language Inference (MNLI), and vice versa for generative tasks. That is the cleanest available evidence that a small decision model should start from an encoder, not from adapting a small decoder.

Candidate backbones

ModernBERT is the default English starting point. Its abstract describes a modernized bidirectional encoder trained on 2T tokens with native 8,192 length, and the Hugging Face (HF) release blog reports base at 149M parameters and large at 395M parameters. The model card adds architectural detail: rotary position embeddings, alternating local-global attention, unpadding, and Flash Attention support, with base at 22 layers and large at 28 layers. The config file for the base checkpoint confirms 768 hidden size, 12 attention heads, 22 layers, and a 50,368 token vocabulary. Hugging Face application programming interface (API) results fetched for this survey show answerdotai/ModernBERT-base with 4,372,504 downloads and Apache 2.0 licensing, which indicates the widest adoption of any backbone considered here. Laya’s English checkpoint builds on ModernBERT-large.

mmBERT is the multilingual counterpart from the same research lineage. Its abstract reports pretraining on 3T tokens of multilingual text in over 1,800 languages, with low-resource languages added during a decay phase. Hugging Face API results show jhu-clsp/mmBERT-base with 400,800 downloads and a smaller mmBERT-small variant. Laya’s multilingual checkpoint builds on mmBERT-base, which Laya’s README describes as covering more than 100 languages and running 2 times faster than its English checkpoint. The Maestro Flash proposal in this graph nominates mmBERT-base as its production backbone candidate for the same reason: no other modern encoder in this size band claims comparable language breadth.

Ettin is a research suite of paired encoders and decoders designed for fair architecture comparison. The paper abstract reports identical data and recipe across sizes from 17M to 1B parameters, and the model card for the 400M encoder reports General Language Understanding Evaluation (GLUE) averages of 88.9 against 88.4 at base scale and 90.8 against 90.4 at large scale, both in Ettin’s favor over ModernBERT, plus Massive Text Embedding Benchmark (MTEB) v2 English retrieval scores of 45.7 against 43.9 at base and 48.4 against 47.0 at large. Hugging Face API results show jhu-clsp/ettin-encoder-400m with 9,660 downloads and MIT licensing. Adoption is therefore far smaller than ModernBERT’s, but Ettin is the only backbone here whose encoder advantage over a matched decoder was measured under controlled conditions. The Maestro Flash proposal names it as its ablation candidate.

DeBERTaV3 represents the older but well-proven disentangled-attention lineage. Its abstract reports replacing masked language modeling (MLM) with replaced token detection (RTD) plus gradient-disentangled embedding sharing, with the large model reaching 91.37 percent average on GLUE. The multilingual variant mDeBERTa base reaches 79.8 percent zero-shot cross-lingual accuracy on cross-lingual NLI (XNLI) according to the same abstract, a 3.6 point improvement over XLM-RoBERTa (XLM-R) base. Microsoft’s DeBERTa GitHub README adds that DeBERTaV3-XSmall uses only 22M backbone parameters while outperforming RoBERTa-base and XLNet-base on MNLI and the Stanford Question Answering Dataset (SQuAD) v2.0, and that a 1.5B DeBERTa reached 89.9 on SuperGLUE against a human baseline of 89.8. Hugging Face API results show microsoft/deberta-v3-base with 2,827,597 downloads. The distinctive asset is the zero-shot NLI classifier ecosystem: Laurer’s paper on NLI-based universal classifiers reports training on 33 datasets with 389 classes, and its abstract notes more than 55 million downloads of older zero-shot classifiers as of December 2023.

XLM-RoBERTa is the legacy multilingual baseline. Its abstract reports training on 100 languages with more than 2 terabytes of filtered CommonCrawl data, with gains of 14.6 percent average accuracy on XNLI, 13 percent average F1 on Multilingual Question Answering (MLQA), and 2.4 percent F1 on named entity recognition (NER) over multilingual BERT, including 15.7 percent on Swahili and 11.4 percent on Urdu XNLI accuracy. It matters here mainly as the reference that newer multilingual encoders must beat, and as the base architecture under BGE-M3.

Retrieval-tuned encoders form a separate family. BGE-M3’s abstract reports support for more than 100 languages, three retrieval functions (dense, multi-vector, and sparse), and inputs up to 8,192 tokens, with Hugging Face API results showing 37,280,275 downloads for BAAI/bge-m3 under MIT license. Nomic Embed’s abstract reports an 8,192 context English embedder that outperforms OpenAI Ada-002 and text-embedding-3-small on short-context MTEB and long-context LoCo benchmarks, and the API shows 200,878 downloads for nomic-ai/modernbert-embed-base. Arctic-Embed 2.0’s abstract reports competitive multilingual and English retrieval with Matryoshka Representation Learning for efficient storage. These are strong starting points for search, but their fine-tuning optimizes embedding geometry for retrieval, not the token-level marker representations that Laya-style and GLiClass-style heads score.

What the evidence says

The table below collects one row per backbone family. Every figure in it is reported by the source named in the rightmost column, and download counts are from Hugging Face API queries run on 2026-09-23. Download counts measure adoption and packaging maturity, not quality.

Several patterns stand out. First, pretraining scale has converged: ModernBERT reports 2T tokens, mmBERT reports 3T tokens, and Ettin reports up to 2T tokens, all far beyond what BERT-era models saw. RoBERTa’s abstract already argued that BERT was significantly undertrained and that training data size and hyperparameters drive results, and the modern suites reflect that lesson. Second, context length has bifurcated. ModernBERT, BGE-M3, and Nomic all report 8,192 native tokens, while DeBERTaV3 and XLM-R remain 512-style models. But native length overstates usable decision context, because Laya truncates ModernBERT to 512 and mmBERT to 1024, and option text shares that budget with state.

Third, the head-to-head numbers favor Ettin over ModernBERT on standard discriminative benchmarks, while DeBERTaV3 posts the highest absolute GLUE figure in this survey. Ettin’s model card reports GLUE 90.8 against 90.4 at large scale and retrieval 48.4 against 47.0, and DeBERTaV3’s abstract reports 91.37 percent GLUE average for its large model. These are not comparable measurements across papers, since recipes, data, and evaluation details differ, so they should be read as existence proofs of competence rather than a ranking. Fourth, multilingual breadth belongs to mmBERT on paper: its abstract claims more than 1,800 languages against XLM-R’s reported 100 and BGE-M3’s reported 100 plus, with the distinctive claim that decay-phase-only low-resource data reaches classification performance similar to large generative models. That last claim is the abstract’s own comparison and has not been independently confirmed in this survey.

Fifth, ecosystem maturity runs in the opposite direction from novelty. ModernBERT-base reports 4.37M downloads, DeBERTa-v3-base reports 2.83M, and BGE-M3 reports 37.28M, while Ettin-encoder-400m reports 9,660 and mmBERT-base reports 400,800. The gap matters for a decision project because downloads proxy for bug reports, fine-tuning recipes, quantization support, and inference integrations. Finally, the zero-shot story belongs to DeBERTaV3 through NLI classifiers, not to raw backbone scores. Laurer’s abstract reports a universal classifier trained on 33 datasets with 389 classes, and the API shows 154,038 downloads for MoritzLaurer/deberta-v3-large-zeroshot-v2.0. No comparable NLI classifier ecosystem was located for ModernBERT or mmBERT in this survey’s fetches.

Analysis

The evidence supports a conditional recommendation, not a single winner. For English-only typed decisions, ModernBERT-large is the sensible default: it combines 2T-token pretraining, 8,192 native context, Flash Attention era efficiency, Apache 2.0 licensing, and the largest adoption footprint in this survey. Ettin-encoder-400M is the correct first ablation because its model card reports small but consistent gains over ModernBERT on GLUE and retrieval, and its controlled encoder-versus-decoder result is directly on point for the architecture decision. If Ettin wins on the decision harness, the project learns that the newer recipe transfers to typed heads. If it loses, the project keeps the mature backbone with no sunk cost beyond one ablation.

For multilingual decisions, mmBERT-base is the only modern option in this size band with reported thousand-language pretraining, which makes it the default despite thinner ecosystem support. The honest concern is verification depth: its headline language count and low-resource claims come from its own abstract, Laya’s multilingual checkpoint reports a 1024 truncation, and this graph’s Laya teardown notes that the multilingual model shipped without fitted temperatures and with mean ECE of 0.314 as self-reported by Laya. A multilingual project should therefore pair mmBERT-base with two controls from day one: an English ModernBERT checkpoint behind a router, to measure what multilinguality costs on the project’s own languages, and continued pretraining or vocabulary checks on the target languages, since machine-translated training text is not equivalent to native text.

DeBERTa-v3-large earns a place as a second ablation in exactly one situation: when zero-shot or few-shot classification over arbitrary labels is the core interaction. Its GLUE and XNLI figures are strong, its 22M XSmall variant shows the recipe scales down, and its NLI classifier ecosystem is the most downloaded zero-shot machinery in this survey. The cost is context and modernity: 512-style length and no reported Flash Attention era efficiency work comparable to ModernBERT’s. For fixed-schema decisions with fine-tuning, that tradeoff favors ModernBERT. For open-label zero-shot, it favors DeBERTaV3, and the harness should decide.

Retrieval-tuned encoders should not be the default backbone even though their download counts dwarf everything else. BGE-M3’s reported 37.28M downloads reflect retrieval adoption, and its abstract optimizes for dense, multi-vector, and sparse retrieval over long documents. A decision head scores option markers inside one sequence, which is closer to GLiClass-style token scoring than to embedding similarity. Starting from a retrieval-tuned checkpoint risks inheriting geometry optimized for a different objective, and any gain would need to be demonstrated against the MLM-pretrained backbone on the decision harness with calibration included. The same reasoning applies to Nomic embed and Arctic-Embed variants. They are candidates for a shortlist or retrieval stage, as Laya’s predict shortlist pattern suggests, not for the trunk.

Two cross-cutting cautions apply to every pick. First, parameter counts are not comparable across accountings: the HF release blog reports ModernBERT-large at 395M while Laya’s README reports its ModernBERT-large checkpoint at 421M, presumably including heads and embeddings counted differently. Compare video memory (VRAM), latency, and accuracy on the same harness instead of comparing headline counts. Second, none of the backbone sources report calibration, option-order robustness, or high-cardinality behavior, which are the exact axes where this graph’s Jev and Laya papers report open problems. The backbone decision must therefore be made on stage-zero harness measurements that include Brier score, ECE, and flip rate, not on GLUE alone.

Implications for small decision models like Jev, Laya, and Velofy’s proposed Maestro Flash

For Jev, the implication is limited by design. Jev is a closed API with undisclosed architecture, as this graph’s Jev survey documents, so no backbone can be attributed to it from public evidence. The behavioral figures in that survey, including third-party reports of Banking77 accuracy, ECE, and option-permutation flip rate, are the only legitimate comparison targets. An open backbone project should treat Jev as a behavioral reference on accuracy, calibration, and latency, and should not claim architectural parity or superiority based on backbone choice.

For Laya, this survey supports the existing backbone pair while sharpening two follow-ups. ModernBERT-large for English and mmBERT-base for multilingual is consistent with the evidence above: the mature English encoder plus the broadest modern multilingual encoder. The follow-ups are the truncation gap and multilingual calibration. Both backbones report long native context, yet Laya truncates to 512 or 1024, so extending usable context for state plus options is the highest-leverage backbone-adjacent change. On calibration, Laya’s own reported multilingual ECE and missing temperatures mean the multilingual trunk needs its own temperature fitting and native-text evaluation before any claim about language coverage becomes a claim about decision quality.

For Velofy’s proposed Maestro Flash, the proposal’s pairing of mmBERT-base with an Ettin-encoder-400M ablation survives this survey, with one amendment. The amendment is to add an English ModernBERT-large control to stage zero even if the production target is multilingual, because that control separates backbone quality from multilinguality cost on every benchmark the proposal lists. The proposal’s day 18 decision between mmBERT and Ettin should be scored on the decision harness (Banking77 high-cardinality choice, ordinal score error, Brier, ECE, median (p50) latency, flip rate) rather than on GLUE or retrieval leaderboards. If mmBERT wins only on language breadth but loses calibration or latency on the project’s core languages, the router pattern Laya already ships is the fallback: route by language instead of forcing one trunk. No training has been done for Maestro Flash, and none of the above predicts which backbone will win.

Limitations

This is a literature and artifact survey, not a benchmark. No model was trained, fine-tuned, or measured for this paper, and no Velofy checkpoint exists. All head-to-head figures come from vendor or author sources: Ettin’s GLUE and retrieval deltas from its own model card, DeBERTaV3’s GLUE and XNLI figures from its own abstract, mmBERT’s language breadth from its own abstract. Independent reproduction on a shared decision harness is the missing step, and cross-paper score comparisons should not be treated as rankings.

Download counts are weak evidence. They were fetched from the Hugging Face API on 2026-09-23 and reflect packaging, age, default recommendations, and retrieval use as much as decision suitability. BGE-M3’s large count does not imply it is a good decision backbone, and Ettin’s small count does not imply it is weak. License tags likewise come from API metadata and model cards and should be rechecked before commercial use.

Several quantities that matter most for decisions are UNVERIFIED at the backbone level: calibration of every pretrained checkpoint, high-cardinality option scoring, option-order flip rate, Indic language decision quality, and latency on the reader’s own hardware. Laya’s T4 figures and this graph’s Apple Silicon smoke test are starting points, not transferable guarantees. Finally, the field is moving: any backbone chosen now should be re-ablated on a fixed harness at least once before production, because the gap between ModernBERT-era and next-generation encoders is measured in months, not years.

Open questions

1. Which backbone wins a fixed stage-zero decision harness that scores Banking77 high-cardinality choice, ordinal error, Brier score, ECE, p50 latency, and option-order flip rate across ModernBERT-large, mmBERT-base, Ettin-encoder-400M, and DeBERTa-v3-large?
2. How much usable decision context can a Laya-style single-pass head actually exploit on ModernBERT-class backbones before option text crowds out state, and does extending truncation from 1024 toward the reported 8192 native length improve high-cardinality accuracy without harming calibration?
3. What is the measured cost of multilinguality on native-text Indic decision benchmarks for mmBERT-base against an English ModernBERT control behind a router, scored per language rather than averaged?

References

1. ModernBERT paper, arXiv:2412.13663
2. mmBERT paper, arXiv:2509.06888
3. Ettin paired encoders and decoders paper, arXiv:2507.11412
4. DeBERTaV3 paper, arXiv:2111.09543
5. DeBERTa paper, arXiv:2006.03654
6. XLM-R paper, arXiv:1911.02116
7. M3-Embedding (BGE-M3) paper, arXiv:2402.03216
8. Nomic Embed paper, arXiv:2402.01613
9. Arctic-Embed 2.0 paper, arXiv:2412.04506
10. NLI universal classifiers paper, arXiv:2312.17543
11. GLiNER paper, arXiv:2311.08526
12. GLiClass paper, arXiv:2508.07662
13. ModernBERT release blog, https://huggingface.co/blog/modernbert
14. ModernBERT GitHub repository, https://github.com/AnswerDotAI/ModernBERT
15. DeBERTa GitHub repository, https://github.com/microsoft/DeBERTa
16. GLiClass GitHub repository, https://github.com/Knowledgator/GLiClass
17. Laya GitHub repository, https://github.com/NandhaKishorM/laya
18. Ettin 400M encoder model card, https://huggingface.co/jhu-clsp/ettin-encoder-400m

← All papers