Usable Decision Context in Single-Pass Encoder Heads

The question

A Laya-style decision head puts option markers, instructions, and state text into one sequence and scores every option in a single forward pass. The backbone underneath reports a long native context, 8,192 tokens for ModernBERT-class encoders, but the decision layer truncates far short of that: Laya’s README reports 512 tokens for its English checkpoint and 1,024 for its multilingual and typed-decisions checkpoints. Options and state share that truncated budget, so adding options necessarily takes tokens away from state, or takes tokens away from each option until labels become indistinguishable.

This report asks how much usable decision context such a head can actually exploit before option text crowds out state, and whether extending truncation from 1,024 toward the reported 8,192 native length improves high-cardinality accuracy without harming calibration. The parent survey is Which Encoder Backbone Should a Small Decision Model Start From?, which frames this as the truncation gap, and the architectural facts come from Laya: An Open Typed-Decision Engine, Examined.

The short answer is that native length is an upper bound, not a usable budget, and no fetched source measures what fraction of it a single-pass decision head can exploit. Laya’s README gives the sharpest available numbers: at default settings 77 Banking77 options receive about 3 to 4 tokens each, and accuracy sits at 0.425 against a third-party Jev figure of 0.870 on 72 labels. Extending truncation should recover label distinctness up to a point, because the README itself recommends raising head and sequence budgets for 50-plus options, but whether accuracy recovers fully, whether state-heavy tasks regress as options grow, and whether calibration survives the change are all UNVERIFIED. No Velofy model has been trained to settle this; the deliverable here is the evidence assembled in one place plus a measurement design that would settle it.

Background

The single-pass layout is fixed by the teardown in this graph. For a choice question, Laya builds a sequence shaped like option markers and instructions first, then state, with one marker vector scored per option and a softmax over the resulting logits. The sequence budget therefore splits two ways: a head budget for the option prompt (head_max_len) and the remainder for state (max_len minus head_max_len). Laya’s README reports the defaults as 512 total with a 192-token head budget on English (about 320 tokens for state) and 1,024 total with a 256-token head budget on multilingual and typed-decisions (about 768 tokens for state). Three consequences follow directly. First, per-option tokens fall as one over the option count. Second, raising the head budget to rescue options shrinks the state budget unless total length rises with it. Third, every per-option token floor quoted below is arithmetic, not a model result, until a harness measures it.

The backbone side of the story is ModernBERT’s alternating attention. The paper abstract reports a modernized bidirectional encoder trained on 2 trillion tokens with a native 8,192 sequence length. The Hugging Face release blog reports base at 149M parameters and large at 395M, with rotary position embeddings (RoPE), alternating local and global attention, unpadding, and Flash Attention support. The base checkpoint config fetched for this report confirms the details: 768 hidden size, 12 heads, 22 layers, 50,368 vocabulary, local attention over 128 tokens, and global attention every 3 layers with a global RoPE theta of 160,000. The mmBERT-base config fetched for this report shows the same ModernBERT-class pattern (22 layers, local window 128, global every 3 layers, theta 160,000) with maximum position embeddings of 8,192 and a 256,000-token vocabulary, which resolves the parent survey’s UNVERIFIED native length for mmBERT: the config reports 8,192. Two facts temper this. Local layers only mix information within 128 tokens, so an option marker and a distant state span meet only at every third layer; and RoPE-based models do not automatically generalize past their trained length, since the YaRN paper abstract states that such models fail to generalize past the sequence length they were trained on. Native 8,192 means the positions exist, not that a fine-tuned decision head uses them well.

The third background piece is position sensitivity inside long contexts. The Lost in the Middle paper abstract reports that language model performance is often highest when relevant information sits at the beginning or end of the input and degrades significantly when models must use information in the middle, even for explicitly long-context models. A single-pass decision sequence has exactly this shape: options at the front, state at the back, and the relevance judgment spanning the middle distance between them. That paper studies generative models on retrieval-style tasks, not encoder decision heads, so it transfers as a caution rather than a prediction. Still, it is the cleanest published reason to doubt that usable context equals native context, and any truncation sweep should test option-first against state-first orderings rather than assuming layout is neutral.

Calibration is mostly orthogonal to context, but the two interact through distribution shift. The standard result, from the Guo et al. calibration paper abstract, is that modern neural networks are poorly calibrated and that temperature scaling, a single-parameter variant of Platt scaling, is surprisingly effective at calibrating predictions. Laya applies this per bucket: its README reports one temperature per question type and option-count bucket (2, 3 to 5, 6 to 10, 11-plus), clamped to [0.5, 5.0] at load. Any truncation change alters the logit distribution the temperatures were fitted on (longer sequences, more options per sequence, different marker-to-state distances), so fitted temperatures should be refit after any context change. Laya’s README already warns that a fallback temperature prevents a loading failure but does not establish calibrated confidence, and its changelog notes that calibration is now fitted on a held-out slice rather than training data. The measurement implication is strict: report post-refit expected calibration error (ECE) and Brier score after every truncation change, never reuse temperatures across lengths.

What the evidence says

The table below collects the context, budget, accuracy, calibration, and latency figures that bear on the question. Every number is attributed to the source that reports it. No figure here was measured by Velofy.

Several patterns stand out. First, the native-versus-usable gap is large and acknowledged. Laya’s README states that the multilingual encoder supports up to 8,192 with RoPE while defaulting to 1,024, and its remediation list for 50-plus options explicitly names raising max_len to 2,048, 4,096, or 8,192. The project therefore treats longer truncation as available but unevaluated, which is exactly the gap this report scopes.

Second, the per-option arithmetic is brutal at defaults and testable in one sweep. At a 256-token head budget with 16 tokens of overhead, 77 options land at 3 to 4 tokens per label. The sibling papers on cardinality (How Many Options Can a Single-Pass Head Handle?) and break points (Where Single-Pass Accuracy Breaks With Option Count) own those axes; this report’s contribution is the context side, namely the two-dimensional sweep over head budget and total length that separates token starvation (fixable by longer truncation) from marker-count limits (not fixable by truncation alone).

Third, the shortlist evidence is suggestive but weak. A top-20 shortlist moving BANKING77 from 54.3 to 60.8 percent comes from a community reporter on their own setup, and Laya’s README explicitly notes it has not remeasured those figures. Shortlist-then-decide is compatible with single-pass heads (Laya ships predict_shortlist; the protocol paper A Shortlist-Then-Rerank Protocol Past Single-Pass Limits owns the design), but the recall tradeoff is UNVERIFIED on Banking77. A truncation study should include a shortlist arm so longer context competes against fewer options rather than being evaluated in a vacuum.

Fourth, latency at current lengths leaves room to test longer truncation. Laya’s README reports 39.5 ms for one English question and 72.3 ms for 10 batched multilingual questions on a T4, and the ModernBERT stack brings Flash Attention 2 and unpadding to keep long sequences cheap. But no fetched source reports decision-head latency at 2,048 or beyond, so the cost curve past 1,024 is UNVERIFIED and must be measured alongside accuracy.

Fifth, calibration after extension is the least evidenced cell in the table. Laya’s 0.081 post-temperature ECE is fitted at current truncations, and per-bucket temperatures absorb option-count effects only within the length regime they were fitted on. Nothing fetched here reports ECE, Brier score, or flip rate for a single-pass head above 1,024 tokens. The default hypothesis should be that refit temperatures recover calibration at each new length, following the Guo et al. recipe, but that is a hypothesis with precedent, not a result.

Analysis

The usable-context question decomposes into three sub-questions with different expected answers. First, how many tokens per option does a label need to stay distinct? The README’s 3-to-4-token floor at 77 options is below what multi-word Banking77 intents need, and raising the head budget from 256 to 512 roughly doubles per-option tokens. A label-length control, short labels against paraphrased long wordings at fixed option count, separates token starvation from marker-count effects: if long wordings recover while short ones were already fine, the binding constraint was tokens per option; if neither recovers, it is the number of markers competing in one softmax. That control is cheap, needs no new data, and should precede any claim about native-length exploitation.

Second, how much state can the head actually use once options stop starving? Here the architecture counsels skepticism about the full 8,192. With local windows of 128 and global mixing only every third layer, fine-grained option-to-state comparisons route through a small number of global layers (about 7 of 22 in base scale, about 9 of 28 at large scale, by division from the reported layer counts). Short state passages of a few hundred tokens sit within a few local windows of the option block and get ample mixing; state spanning thousands of tokens relies on the global layers to carry distant evidence back to the markers. The Lost in the Middle caution applies in this middle distance: even models that accept long inputs use their edges better than their middles. None of this proves a decision head fails at 4,096 or 8,192, but it predicts diminishing returns with state length, and possibly a state-ordering effect if evidence sits mid-passage. The sweep should therefore vary state length independently of option count (short, medium, and long state bands at fixed option counts) and report accuracy per band rather than pooling.

Third, does extending truncation harm calibration? Mechanically, longer sequences change the softmax inputs: more markers compete, marker representations shift as they attend over longer state, and any length-correlated confidence bias in the trunk leaks into the logits. Laya’s per-bucket temperature scheme already concedes that calibration depends on option count; length is plausibly a second grouping variable. The conservative analysis is that post-refit calibration at each length should match current figures if the accuracy signal is intact, because temperature scaling corrects exactly the global confidence scale that length shift perturbs. The failure mode to watch is not average ECE but conditional miscalibration: confident errors on crowded sequences, like the README’s Khmer 0.000 accuracy at 0.952 confidence (a language failure, not a length failure, but the same confident-wrong shape). The sweep should report ECE sliced by sequence-length quartile and by per-option token floor, plus flip rate under option permutation at each length, since longer option blocks give position bias more room to act. Laya’s README already documents position-adjacent pathologies (a multilingual score bias against the first-listed level, an English noul label-dominance effect), which makes permutation testing at each new length mandatory rather than optional.

Putting the three together yields a concrete measurement design, which is this report’s main constructive output. Fix one backbone (ModernBERT-large for English, matching Laya’s English trunk) and one head implementation. Sweep max_len over 512, 1,024, 2,048, 4,096, and 8,192 with head_max_len scaled to hold per-option tokens constant in one arm (isolating state-length effects) and held fixed in another arm (reproducing the crowding regime). At each cell, report Banking77 accuracy, a short-label against long-label control, ordinal error on a score task, Brier score, post-refit ECE with refit done per cell on held-out data, option-permutation flip rate, and median (p50) latency on fixed hardware. Include a shortlist arm (top-20 bi-encoder shortlist, then one forward pass) and a state-first ordering arm at two lengths. Stop extending when accuracy per band flattens or calibration per cell degrades after refit; that plateau is the measured usable context, and it will almost surely sit below 8,192. The stage-zero harness paper in this graph (A Stage-Zero Backbone Harness for Typed Decisions) already specifies the six-axis scoring this sweep should reuse, so the marginal cost is the length sweep itself.

A final analytical note concerns what longer truncation cannot fix. If the break point tracks head design (marker count, single-layer scoring, softmax competition) rather than token budget, then 8,192 tokens of room will not move it, and the fix is architectural: decoupled option encoders, hierarchical coarse-to-fine decisions, or shortlist-then-decide pipelines. The constant-per-option arm of the sweep is the diagnostic: if accuracy still breaks at the same option count when every option has ample tokens, the ceiling is the head, not the context. That outcome would redirect effort from truncation tuning toward the head designs the sibling papers scope, which is a useful negative result, not a failure.

Implications for small decision models like Jev, Laya, and Velofy’s proposed Maestro Flash

For Jev, the implication is comparative only. Jev is a closed application programming interface (API) with undisclosed architecture, as documented in Jev and the Typed-Decision Landscape, so no context mechanism can be attributed to it from public evidence. The usable behavioral facts are its reported 255-choice cap, the third-party Banking77 figure of 0.870 on 72 labels quoted in Laya’s README, and the independently measured ECE of 0.246 and 13 percent permutation flip rate reported by the decision-model benchmark surveyed in the Jev paper. An open truncation study should treat those as the behavioral bar: if extended context plus refit calibration reaches Jev-class high-cardinality accuracy at Laya-class latency (tens of milliseconds on a T4 against Jev’s independently measured 236 to 276 ms p50), the truncation gap was the story; if accuracy recovers but calibration or flip rate does not, the gap is the head or the training recipe. No architectural claim about Jev follows either way.

For Laya, the implications are immediate and practical. The README already documents the token-budget constraint, the remediation knobs, and the shortlist fallback, so this report adds measurement discipline rather than new machinery: sweep the knobs the project already ships, refit temperatures per cell on held-out data, and publish accuracy with ECE, Brier, flip rate, and latency at each length instead of accuracy alone. Two specific cautions come from the README’s own honest-limits section. First, noul label dominance and multilingual score position bias are both option-prompt pathologies, and longer option blocks may amplify them; permutation and ordering arms at each length are therefore part of the sweep, not extras. Second, action.act_probability carries no usable signal (area under the receiver operating characteristic curve, AUROC, of 0.30 against correctness on 396 labelled decisions, per the README) while entropy confidence reaches 0.77, so gating during the sweep should use confidence, and the action head should be left out of the usable-context story until it is retrained.

For Velofy’s proposed Maestro Flash, the implication is a sequencing decision for stage zero. The proposal nominates mmBERT-base with an Ettin ablation and a ModernBERT control; this report adds that the harness must sweep truncation as a first-class variable from day one, because backbone comparisons at a single pinned length confound trunk quality with budget adequacy. Concretely: run the 512-to-8,192 sweep on the ModernBERT-large control first, since its 8,192 native length and Laya lineage make it the reference trunk, then repeat the winning lengths on mmBERT-base and Ettin rather than sweeping all trunks blindly. If usable context plateaus early (say, past 2,048) while the per-option arm keeps improving, production should size max_len at the plateau and spend the remaining budget on shortlisting or head design, not on longer sequences. Nothing here predicts where the plateau lands, and no Maestro Flash training has been done; the point is that the harness must be built to find it.

Limitations

This is a literature and artifact report, not a benchmark. No model was trained, fine-tuned, or measured for this paper, no Velofy checkpoint exists, and every figure is attributed to the source that reports it. The central quantities (usable context length, post-extension accuracy, post-refit calibration above 1,024 tokens, latency past 1,024 tokens) are UNVERIFIED because no fetched source measures a single-pass decision head above its default truncation.

Several attributions are weaker than they look. The shortlist improvement (54.3 to 60.8 percent) is a community reporter’s figure that Laya’s README explicitly has not remeasured. Jev’s 0.870 Banking77 figure is quoted secondhand in Laya’s README from third-party work, on 72 labels rather than 77, and the Jev survey in this graph documents the benchmark-to-benchmark variance. Latency figures describe specific hardware and packaging, not transferable guarantees; the T4 numbers come from Laya’s README and do not predict other accelerators. The Lost in the Middle and YaRN results concern generative models and RoPE extrapolation generally, not encoder decision heads, so they transfer as cautions with mechanistic plausibility, not as predictions with error bars.

Finally, the measurement design assumes the Laya-style joint encoding pattern (options and state in one sequence). Decoupled designs that pre-encode options, like bi-encoder shortlists or label-embedding heads, face a different context tradeoff in which state alone fills the sequence, and this report’s sweep does not cover them. The sibling papers on cardinality and shortlisting own that comparison.

Open questions

1. What is the measured usable context of a Laya-style single-pass head on ModernBERT-large: at what max_len between 512 and 8,192 does Banking77 accuracy per state-length band flatten, and does post-refit ECE stay flat or degrade at each step?
2. Does a label-length control separate token starvation from marker-count ceilings: do long paraphrased wordings recover with head-budget relief while the option-count break point stays fixed, or do both move together?
3. At what option count and state length does a top-k shortlist followed by one forward pass beat full-set single-pass scoring on accuracy per millisecond at matched post-refit ECE, and does that crossover move as truncation extends?

References

1. ModernBERT paper, arXiv:2412.13663
2. mmBERT paper, arXiv:2509.06888
3. GLiClass paper, arXiv:2508.07662
4. GLiNER paper, arXiv:2311.08526
5. Efficient Intent Detection with Dual Sentence Encoders (Banking77 dataset), arXiv:2003.04807
6. Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172
7. On Calibration of Modern Neural Networks, arXiv:1706.04599
8. Longformer: The Long-Document Transformer, arXiv:2004.05150
9. YaRN: Efficient Context Window Extension of Large Language Models, arXiv:2309.00071
10. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, arXiv:2205.14135
11. M3-Embedding paper, arXiv:2402.03216
12. Nomic Embed paper, arXiv:2402.01613
13. Laya GitHub repository, https://github.com/NandhaKishorM/laya
14. GLiClass GitHub repository, https://github.com/Knowledgator/GLiClass
15. ModernBERT GitHub repository, https://github.com/AnswerDotAI/ModernBERT
16. ModernBERT release blog, https://huggingface.co/blog/modernbert
17. ModernBERT-base config, https://huggingface.co/answerdotai/ModernBERT-base
18. mmBERT-base model card, https://huggingface.co/jhu-clsp/mmBERT-base

← All papers