Option Cardinality Limits in Small Decision Models

The question

This survey asks why small typed-decision models lose accuracy as the option set grows, and why two limits keep appearing in public materials. Laya’s BENCHMARKS.md advises keeping choice questions under about 20 options, and reports that both base checkpoints score 0.425 on Banking77, a dataset that Casanueva et al. describe as covering 77 intents over 13083 examples. TypeSafe states that Jev supports cardinality up to 255, and adds that higher-cardinality choices use a two-stage system that scores independently and then makes an explicit choice. The independent decision-model benchmark (DMB) by nibzard reports that Jev answers its code-word suite at 254 and 255 options with 100 percent accuracy, then rejects 256, 384, and 512 options with 400 Too many choices, while every large language model (LLM) it tested handles 512 options.

The soft guidance and the hard cap are different failures. The soft guidance marks gradual degradation: jointly encoded options become harder to tell apart. The hard cap marks a sharp refusal: the request never reaches scoring. This paper reports no new model results. Velofy has not trained or shipped a decision model, and every Maestro Flash statement below is a proposal target.

First principles

Four constraints explain most of what follows. Each follows from how a single-pass decision model is built and served, before any benchmark is consulted.

A shared budget divides by the number of options. A joint encoder reads instructions, every option, and the state as one sequence with a fixed maximum length. If the options share a fixed slice of that sequence, each added option shrinks every other option’s share. Enlarging the slice takes tokens from the state instead. At any given maximum length the total is fixed, so the trade can be moved but not avoided.

A softmax spreads one unit of probability over every option. The model returns one probability per option, and they sum to one. The random baseline is one divided by the option count: 0.5 at two options, 0.05 at twenty. The same raw scores give a lower top probability when more options share the mass, so a confidence scale fit on small sets does not carry over unchanged to large ones.

More options means more positions and more orderings. In one ordered sequence, each option has a position and neighbors, and both change when the options are reordered. Nothing in a standard Transformer forces the scores to ignore that, so order sensitivity has more room to appear as the list grows.

A hard cap is a rule, not a slope. The three constraints above predict a gradual loss of accuracy. A request refused at one exact count points to a designed limit, such as a validation rule or a table sized in advance, not to gradual confusion. The two need different evidence and different fixes.

Background

Typed decisions pose classification as an inference-time question. The caller supplies state plus a label set, and the model returns a probability vector in one pass. Laya’s README describes choice, score, and noul primitives answered in a single forward pass, at 33 ms for one question and 7.2 ms per question batched on a T4. The parent Laya teardown in this graph reads Laya’s code as building one sequence from instructions plus masked option markers plus state, then scoring each marker with softmax. The parent Jev survey covers Jev behavior and alternatives. This paper does not repeat those reports and focuses only on what changes as labels grow.

Three encoding families matter. Joint encoding places state and all options in one sequence, so every option attends to the state and to every other option, but the representation budget grows with the label set. Separate (dual) encoding embeds state and each option independently, then scores with a dot product or cosine similarity. Casanueva et al. describe dual sentence encoders for intent detection, Reimers and Gurevych describe siamese BERT networks for practical pairwise comparison, and Karpukhin et al. describe a dual-encoder retriever that outperforms a Lucene BM25 baseline. Tunstall et al. describe SetFit as contrastive siamese fine-tuning followed by a classification head, with no prompts or verbalizers. Cross encoding scores each text-label pair separately with full attention. The GLiClass abstract states the bottleneck directly: cross encoders must process text-label pairs sequentially, which reduces efficiency with large label sets.

Context length alone does not settle the question. Warner et al. describe ModernBERT as trained on 2 trillion tokens with a native 8192 sequence length. Laya’s README states that Laya defaults to 512 context with head_max_len 192, and that multilingual checkpoints default to 1024 context with head_max_len 256, even though, as the README notes, the mmBERT base encoder supports up to 8192 with rotary position embeddings (RoPE). The binding constraint is therefore a configured split between option budget and state budget, not the backbone maximum. Under defaults the README states about 320 state tokens on English and about 768 on multilingual, so raising the option budget directly shrinks state context.

Two findings predict trouble before any hard cap. Liu et al. report that models use long contexts unevenly, with best performance when relevant information sits at the beginning or end and worse performance in the middle, on multi-document question answering and key-value retrieval. Zheng et al. report selection bias on multiple-choice questions, a systematic preference for specific option identifiers across 20 models and three benchmarks, and propose estimating that prior by permuting option contents.

What the evidence says

All Jev behavior numbers below are third-party or vendor-stated, since no Jev weights or training paper are public. All Laya numbers are self-reported unless noted. Cross-project comparison is approximate because prompts, samples, and label counts differ.

Laya documents a token-budget mechanism in unusual detail. Its README states that options share a fixed head_max_len budget, 192 tokens on English and 256 on multilingual, so 77 options receive only about 3 to 4 tokens per label and become indistinguishable. The README gives the arithmetic as (256 - 16) / 77. Laya’s BENCHMARKS.md reports Banking77 accuracy of 0.425 for both base checkpoints and 0.492 for the typed-decisions checkpoint, against a published Jev figure of 0.870 that it cites from third-party work. It notes that both base checkpoints scoring exactly 0.425 looks like a budget ceiling rather than a capability gap, and advises keeping choice questions under about 20 options. The README states that Jev is better suited for 50 or more options in a single prompt without tuning.

Laya also documents order sensitivity at that scale. BENCHMARKS.md reports flip rates on MASSIVE intent English of 0.150 for Laya and 0.230 for multilingual, and cites a measured Jev figure of 0.13. On English emotion it reports 0.040 and 0.090, and on XNLI English 0.000 and 0.015. It describes MASSIVE intent as using 20 options with a random baseline of 0.050, and suggests more aggressive option shuffling during training as a fix. It reports median (p50) latency of 39.5 ms and 32.8 ms on a T4, with 103 to 332 batched questions per second.

AbdelStark provides a direct Jev-versus-encoder pilot. It reports AG News with 4 labels at Jev 0.910 versus GLiNER 0.700, and Banking77 BTZSC with 72 labels at Jev 0.870 versus GLiNER 0.610. On DAIR Emotion with 6 labels, accuracy is unresolved at 0.480 versus 0.440, but Jev is worse calibrated: Brier 0.846 versus 0.668, negative log-likelihood (NLL) 5.588 versus 1.381, and zero probability on the true label for 16 percent of examples. The pilot uses 300 held-out examples with 100 per condition, and notes that its pinned BTZSC configuration exposes 72 hypotheses, with 200 rows that have no positive candidate excluded. That 72-label detail matters: DMB reports a 77-way task and Laya reports 77 labels, so the Banking77 figures do not share a denominator. On latency, AbdelStark reports GLiNER faster on small-label tasks, at about 44 ms p50 versus Jev at 236 to 256 ms, and Jev slightly faster on the 72-label condition, at 246 ms versus 296 ms p50. GLiNER ran locally on an Apple M4 Max central processing unit (CPU) while hosted Jev was called from France, so these are operational measurements rather than normalized timings.

DMB adds the sharp cap and the flat latency inside it. It reports Jev at 76.3 percent on 77-way banking, behind gpt-oss-120b at 81.3 percent. It reports Jev p50 latency of 264 to 276 ms, flat from 2 to 255 options, which it describes as 1.2 times faster than gpt-oss-120b on Cerebras and 10 to 16 times faster than thinking-mode models. On its code-word suite it reports 100 percent accuracy at 254 and 255 options, then 400 Too many choices at 256, 384, and 512, while every LLM handles 512 options. DMB reports an expected calibration error (ECE) of 0.246 for Jev against an LLM range of 0.039 to 0.122, and reports that Jev changes 13 percent of choices under permutation while the worst LLM changes 37 percent. DMB describes five suites and 60 cells with published raw logs, and states that its banking suite uses the PolyAI banking77 dataset under the Creative Commons Attribution 4.0 license (CC BY 4.0).

TypeSafe confirms the ceiling. It states support up to 255, with higher-cardinality choices using a two-stage system of independent scoring then explicit choice, which TypeSafe says causes occasional slowdown. The internal mechanism remains UNVERIFIED. The observed behavior is validation refusal, not a gradual slope.

The open ecosystem shows the same tradeoff elsewhere. The GLiClass README states single-forward-pass classification and reports results about 10 times faster than cross-encoder models. It documents hierarchical labels with dot notation, though no fetched GLiClass source reports a measured curve of accuracy against label count. The Hugging Face application programming interface (API) reports 243120 downloads for fastino/gliner2.5-multi-v1, an adoption signal rather than quality evidence. Reimers and Gurevych describe reducing a pairwise search over 10000 sentences from about 50 million computations and about 65 hours with BERT to about 5 seconds with Sentence-BERT. Karpukhin et al. report that their dense dual encoder outperforms Lucene BM25 by 9 to 19 percent absolute in top-20 passage retrieval accuracy. Both results show why decoupled embeddings scale: option vectors do not compete for one joint context.

Analysis

Four mechanisms explain soft degradation, and a separate system choice explains the hard cap.

First, per-option token starvation is directly documented. When options share a fixed budget, each added label takes tokens from the others. Laya’s README gives 192 tokens on English and 256 on multilingual, with (256 - 16) / 77 yielding about 3 to 4 tokens per label. At that density, labels that differ by one content word can collapse to the same truncated prefix. The README states two remedies. One is to raise head_max_len to 512 and max_len to 1024, or up to 2048, 4096, or 8192. The other is to shortlist with embeddings and run one forward pass on the top k labels. The first remedy spends state tokens, since the README states about 320 state tokens on English and about 768 on multilingual under defaults. The second adds a retrieval stage whose misses never reach the scorer. Neither is free.

Second, position effects grow with option count even when tokens fit. Liu et al. report weaker use of information in the middle of long contexts. Zheng et al. report a preference for specific option identifiers, attributed to token bias. Laya’s BENCHMARKS.md reports flip rates of 0.000 and 0.015 on XNLI, rising to 0.150 and 0.230 on 20-option MASSIVE intent. DMB reports 13 percent flips for Jev and up to 37 percent for the worst LLM.

Third, calibration drifts with option count unless it is parameterized by option count. Laya’s BENCHMARKS.md reports as-shipped ECE of 0.466 and 0.314, with a temperature refit bringing ECE to 0.081 and 0.106, and describes refitting one temperature per question type and option count as the highest-value fix. That grouping admits that one global temperature does not transfer across small and large sets. As the softmax denominator gains terms, the same logits yield lower top probabilities, while entropy-based confidence must normalize by option count to stay comparable. A model fit mostly on small sets will be miscalibrated on large sets even if ranking holds. DMB reports Jev ECE of 0.246 against an LLM range of 0.039 to 0.122, which shows that flat latency inside the cap does not imply flat calibration. Any cardinality claim that reports only accuracy is incomplete. The needed curve is accuracy, ECE, and flip rate against option count.

Fourth, a mismatch in the training distribution is likely but UNVERIFIED as a distinct cause. No fetched source discloses the option-count histogram used to train Jev or the Laya base checkpoints. If most training questions use few options, then a Banking77-scale test is out of distribution in marker count, head length, and softmax size. Laya’s BENCHMARKS.md reports base checkpoints at 0.362 and 0.342 on its typed-decisions set, against a majority baseline of 0.461 and a random baseline of 0.318, with the headline 0.766 coming only after fine-tuning on that training split. That pattern fits narrow training coverage, but it does not isolate option count from domain or wording. A clean test would fix the state text and vary only the labels, which no fetched source reports.

The hard cap is a different fact. DMB reports full success at 254 and 255 and refusal at 256 and above. TypeSafe states support up to 255 and describes a two-stage path for higher-cardinality choices. Together these suggest a designed ceiling with a staged path beneath it. Why that specific cap value was chosen is UNVERIFIED. Candidates include an identifier range with one reserved slot, a fixed output head sized in training, a positional table sized in training, or a product validation rule matching the largest tested configuration. Behavior alone cannot distinguish them. The flat latency from 2 to 255 that DMB reports is consistent with fixed allocation beneath the cap, while the occasional slowdown TypeSafe mentions is consistent with a second stage on larger sets within the cap. Both are consistency arguments, not identifications.

Architectures that avoid these limits share one property: they keep per-option cost from consuming state context. Decoupled option encoding embeds labels once, caches static vectors, and scores state against the cache. Adding labels costs little at inference, at the price of losing cross-option attention. Retrieve-then-rerank keeps joint scoring but bounds its input: an embedding shortlist selects the top k candidates, then one joint pass scores only that shortlist. Laya’s README documents this as predict_shortlist with a caller-supplied embedding function, and cites issue 102 as moving a Banking77 run from 54.3 percent to 60.8 percent with a top-20 shortlist, while noting the repository has not remeasured those figures. Hierarchical decomposition splits one flat choice into smaller grouped choices, which the GLiClass README supports with dot-notation labels. Independent scoring then explicit choice, which TypeSafe describes as its higher-cardinality path, scores options without full cross-attention and then resolves a shortlist. All four trade some cross-option reasoning for bounded context and predictable cost.

Implications for small decision models

For Jev, the product is already a two-regime system. Inside the cap, DMB reports flat latency and full code-word accuracy to 255, while AbdelStark reports 0.870 on 72-label Banking77 BTZSC. That explains why Laya’s README states Jev suits 50 or more options without tuning. Outside the cap the API refuses, so callers above the stated cap must shard labels, shortlist on the client side, or reconcile multiple calls, and the fetched vendor post documents none of those patterns. Calibration inside the cap needs measurement: DMB reports ECE 0.246, and AbdelStark reports zero probability on the truth for 16 percent of DAIR Emotion examples. Teams should measure accuracy, ECE, and flip rate against option count on their own labels rather than extrapolate from small-label AG News, where AbdelStark reports Jev at 0.910, to a Banking77-scale intent task.

For Laya, the limit is transparent and tunable, but the tuning burden is real. Laya’s README states the head budgets and the arithmetic that yields 3 to 4 tokens per label at 77 options, and the escapes: a larger head_max_len and max_len, or an embedding shortlist. Larger head values truncate state on long documents, since the README states about 320 and about 768 state tokens under defaults. The shortlist evidence is the reporter figure of 54.3 percent to 60.8 percent with a top-20 shortlist, which the README marks as not remeasured. BENCHMARKS.md advises keeping choice under about 20 options at defaults, and reports flips of 0.150 and 0.230 at 20 options. The practical reading is narrow. Laya is predictable below the soft threshold, diagnosable at it, and experimental above it until shortlist recall and calibration are measured on caller data.

For Velofy’s proposed Maestro Flash, which has no weights and no results, the implication is a test plan. The companion proposal suggests separately encoded, cacheable option representations so cardinality does not consume state budget, plus temperatures per question type, option count, and language. The tests that would validate or refute that plan are concrete. Sweep the Banking77 option count up to the full 77 intents that Casanueva et al. describe, with fixed state text and only the labels varying. Report ECE and flip rate on the same sweep, not just accuracy. Compare joint encoding against decoupled encoding and shortlist-then-rerank under a matched backbone and data, with shortlist recall reported separately so retrieval misses are visible. Test hierarchical labels on larger taxonomies, since no fetched source here measures that regime for typed-decision encoders. Until those curves exist, high-cardinality fixes remain hypotheses. This survey proposes no new architecture.

Limitations

Jev internals are UNVERIFIED. Every Jev number here comes from vendor claims or two third-party projects with their own prompts and samples. AbdelStark reports 300 held-out examples with 100 per condition, which supports pilot intervals but not per-class claims on a 72-label task. DMB describes five suites and 60 cells with raw logs, but DMB evaluates 77-way banking while AbdelStark evaluates 72 hypotheses after excluding 200 rows with no positive candidate, so the two Banking77 figures do not share a denominator. Laya figures are self-reported and were read from its README and BENCHMARKS.md, not reproduced here. The shortlist gain from 54.3 percent to 60.8 percent is a reporter figure that Laya’s README marks as not remeasured. Latency comparisons are deployment-specific, and download counts, such as the 243120 downloads the Hugging Face API reports for a GLiNER checkpoint, measure reuse rather than quality. Multilingual high-cardinality evidence is thin. Laya’s BENCHMARKS.md reports MASSIVE intent at 20 options across 51 languages, but no fetched source here reports Banking77-scale sets outside English. No experiment was run for this survey.

Open questions

1. At what option count does decoupled option encoding surpass joint encoding on Banking77 when backbone, training data, and calibration budget are matched?
2. How should temperature or calibration parameters be grouped by option count so that ECE and flip rate stay flat as label sets grow?
3. What shortlist-then-rerank protocol preserves Banking77 accuracy and calibration past single-pass limits while reporting shortlist recall separately?

References

1. TypeSafe, Introducing System One Models and Jev
2. Laya README, NandhaKishorM/laya
3. Laya BENCHMARKS.md, NandhaKishorM/laya
4. nibzard/decision-model-benchmark
5. AbdelStark/jev-benchmarks
6. Casanueva et al., Efficient Intent Detection with Dual Sentence Encoders, arXiv:2003.04807
7. Warner et al., ModernBERT, arXiv:2412.13663
8. Marone et al., mmBERT, arXiv:2509.06888
9. Zaratiana et al., GLiNER, arXiv:2311.08526
10. Stepanov et al., GLiClass, arXiv:2508.07662
11. Reimers and Gurevych, Sentence-BERT, arXiv:1908.10084
12. Karpukhin et al., Dense Passage Retrieval, arXiv:2004.04906
13. Liu et al., Lost in the Middle, arXiv:2307.03172
14. Zheng et al., Large Language Models Are Not Robust Multiple Choice Selectors, arXiv:2309.03882
15. Tunstall et al., SetFit, arXiv:2209.11055
16. GLiClass README, Knowledgator/GLiClass
17. GLiNER README, Knowledgator/GLiNER
18. Hugging Face API, fastino/gliner2.5-multi-v1

← All papers