Sub-1B Decoders as Decision Models: Constrained Generation Against Encoders

Status: SURVEY. This paper synthesizes published work, public model cards, and open-source documentation. It reports no new experiments and no Maestro Flash results, since Maestro Flash has not been trained. All numeric claims below are attributed to the source they came from. Claims that could not be verified in a fetched source are marked UNVERIFIED.

This paper is a child of the parent survey on typed decisions without generation, which defined the typed-decision category and argued that purpose-built non-generative architectures fit it better than prompted decoders. That paper is assumed background and is not repeated here. This paper asks the follow-up the parent left open: does the case against decoders still hold when the decoder is small (under one billion parameters) and its output is constrained by a grammar, regex, or fixed choice list?

The question

The parent survey gave three reasons to prefer non-generative architectures for typed decisions: latency (one forward pass per generated token), calibration (token-product probabilities are awkward as label beliefs), and parseability (generated text must be parsed back into a typed value). Two developments qualify that picture enough to deserve their own analysis.

First, the sub-1B decoder class has matured into a crowded field with public weights and public evaluation cards: SmolLM2 at 135M, 360M, and 1.7B parameters, Qwen3 starting at 0.6B, Qwen2.5 at 0.5B, and Gemma 3 with a 270M variant. These models run locally on modest hardware, which changes the latency and cost arithmetic the parent computed against hosted application programming interfaces (APIs).

Second, constrained decoding has matured into standard infrastructure. Libraries now guarantee the structure of generated text during generation rather than repairing it afterward, and serving engines offer decoding constrained to a fixed choice list or to JavaScript Object Notation (JSON) as first-class options. If parseability was one of the three counts against decoders, constrained decoding claims to settle exactly that count.

The question: on typed decisions (pick one of k labels, produce a bounded score, return a calibrated probability, fill a typed schema), how do sub-1B decoders with constrained decoding compare with encoder classifiers? In short, no located public study runs this exact head-to-head, so the answer must be assembled from adjacent evidence: matched-size encoder-versus-decoder training studies, small-decoder evaluation cards, constrained-decoding engine papers, and third-party decision benchmarks that tested large decoders rather than small ones. Each piece is attributed; the gaps are named.

Background

Encoder background (ModernBERT, mmBERT, Laya, GLiNER) is covered in the parent survey and the Laya teardown, so this section covers only the decoder side plus the one mechanism that matters.

The sub-1B decoder field. SmolLM2 is documented as a data-centric small-model family, with its 1.7B flagship trained on about 11 trillion tokens and reported to outperform Qwen2.5-1.5B and Llama3.2-1B (SmolLM2 abstract, arXiv:2502.02737). Its public cards extend the family downward to SmolLM2-135M and SmolLM2-360M, both Transformer decoders with published zero-shot numbers. Qwen3 spans 0.6B to 235B parameters, unifies thinking and non-thinking modes, and reports multilingual support expanded from 29 to 119 languages and dialects (Qwen3 abstract, arXiv:2505.09388); the Qwen3-0.6B card lists 0.6B total parameters and 0.44B non-embedding parameters. Gemma 3 spans 1B to 27B parameters with at least 128K tokens of context (Gemma 3 abstract, arXiv:2503.19786), with a 270M variant published on Hugging Face. Qwen2.5-0.5B is the 0.5B predecessor the SmolLM2 cards use as a comparison point. This yields a real 135M to 600M population overlapping the 322M to 421M band Laya reports for its encoder checkpoints, which makes matched-size comparison meaningful.

Constrained decoding mechanics. The core idea, as formulated in the Outlines line of work, is to reformulate generation as transitions between the states of a finite-state machine, build an index over the vocabulary, and thereby guide generation with regular expressions (regexes) and context-free grammars (CFGs) with little added overhead (arXiv:2307.09702). The Outlines README states the consequence directly: it guarantees structured outputs during generation. XGrammar refines the mechanism by splitting the vocabulary into context-independent tokens that can be prechecked and context-dependent tokens checked at runtime, and its paper reports up to 100x speedup over existing solutions with near-zero overhead end to end (arXiv:2411.15100); its README claims 100% structural correctness and lists integrations with vLLM, SGLang, TensorRT-LLM, and MLC-LLM. Complementary interfaces exist at other layers: llama.cpp constrains outputs with GGML Backus-Naur Form (GBNF) grammars, for example to force valid JSON (llama.cpp grammars guide); Guidance constrains generation with regex and CFGs and states that this reduces latency and cost relative to conventional prompting (Guidance README); vLLM exposes structured outputs including choice, JSON, regex, and grammar modes (vLLM documentation). An older structured-decoding line with a conditional random field (CRF) module reported 8 to 14ms of added latency for large consistency gains (arXiv:1910.11555).

First principles

Three mechanical facts govern everything below.

Constraints do not change the token loop. Constrained decoding restricts which tokens are allowed at each step. It does not change the autoregressive factorization: the model still pays one forward pass per emitted token. A choice constraint over single-token labels costs roughly one forward pass, comparable in shape to an encoder pass at similar size, while a JSON object still costs one pass per token. Constraint engines have made per-step masking overhead negligible; they have not removed the per-step forward pass.

Constraints change what is legal, not what is preferred. The mask removes invalid tokens and leaves the model’s ranking of the valid ones untouched. The engine can guarantee format. Correctness still has to come from the weights.

A decoder’s label probability is assembled, not emitted. Under guided choice, a label’s score is the next-token probability renormalized over the allowed tokens, and a multi-token label multiplies several such factors. An encoder head emits one softmax over the labels directly. The two numbers mean different things until each is calibrated on held-out data.

At matched size, the two architectures spend the same parameters differently. An encoder attends in both directions and is usually pretrained to fill in masked tokens. A decoder attends only to earlier tokens and is pretrained to predict the next one. Which spend suits a fixed-label decision is an empirical question, and Ettin’s matched-size study, below, is the cleanest test of it.

What the evidence says

Four bodies of evidence bear on the question. None is the head-to-head study this paper wants. Together they bound the answer.

First, matched-size architecture evidence favors encoders for classification-shaped tasks. The Ettin suite trains paired encoder-only and decoder-only models from 17M to 1B parameters on up to 2 trillion tokens with the same recipe for both, and reports beating ModernBERT as an encoder and Llama 3.2 and SmolLM2 as decoders at matched sizes (arXiv:2507.11412). Its findings are that encoders excel at classification and retrieval while decoders excel at generative tasks, that a 400M encoder outperforms a 1B decoder on Multi-Genre Natural Language Inference (MNLI), and that adapting a decoder to encoder tasks through continued training is subpar compared with training natively. This is the strongest located evidence on the architectural question, and it cuts against small decoders at matched size. Two caveats apply: MNLI is one task, not the typed-decision category with its dynamic option sets and calibration-gated automation; and Ettin compares native pretraining objectives, not a small decoder with constrained decoding plus task-specific fine-tuning.

Second, small-decoder cards show weak zero-shot decision-adjacent ability that improves steeply with size and instruction tuning. The SmolLM2-135M card reports HellaSwag 42.1, AI2 Reasoning Challenge (ARC) average 43.9, and Massive Multitask Language Understanding (MMLU) cloze 31.5 for the base model (lighteval, zero-shot), and instruction-following evaluation (IFEval) 29.9, ARC 37.3, and MMLU cloze 29.3 for the instruct model. The SmolLM2-360M card reports HellaSwag 54.5 against Qwen2.5-0.5B at 51.2, ARC average 53.0 against 45.4, MMLU cloze 35.8 against 33.7, and Grade School Math 8K (GSM8K) 3.2 against Qwen2.5-0.5B at 33.4, with the instruct comparison showing IFEval 41.0 against 31.6. MMLU cloze scores around 30 to 36 sit far below fine-tuned classifier performance, suggesting limited zero-shot headroom at this size. Instruction following (IFEval 29.9 at 135M, 41.0 at 360M) is poor in absolute terms, and it matters because constrained decoding enforces format, not correctness: the model must still select the right option. And the GSM8K inversion shows small-model rankings are task-sensitive, so extrapolating from any single generative benchmark to typed decisions is unreliable. No located card reports a typed-decision suite (dynamic choice sets, rubrics, expected calibration error (ECE)) for a sub-1B decoder.

Third, constrained-decoding overhead is negligible, but sub-1B decision latency and accuracy are unmeasured. XGrammar reports up to 100x speedup over existing solutions with near-zero overhead in end-to-end serving (arXiv:2411.15100). That settles the masking cost. What no located public source provides is the per-decision latency and accuracy of a named sub-1B decoder (for example SmolLM2-360M or Qwen3-0.6B) under guided choice on a fixed typed-decision suite at fixed hardware. Laya’s README gives the encoder reference (32.8ms single, 7.2ms per question batched on an NVIDIA T4 graphics processing unit (GPU)), but no counterpart decoder numbers on the same suite and hardware were located. This cell is UNVERIFIED, and it is the single most decision-relevant missing measurement in this paper.

Fourth, third-party decision benchmarks tested large decoders, not small ones, and their results should not transfer downward without evidence. The decision-model benchmark reports Jev at 76.3% on Banking77 77-way classification behind gpt-oss-120b at 81.3% and glm-5.3 at 80.4%, Jev median (p50) latency 264 to 276ms flat across 2 to 255 options (1.2x faster than gpt-oss-120b on Cerebras, 10 to 16x faster than thinking-mode models), Jev ECE 0.246 against large language models (LLMs) at 0.039 to 0.122, a 13% option-permutation flip rate, $0.07 per 1,000 decisions against $0.19 for the cheapest LLM, and a hard 255-choice cap with rejection at 256 and above (nibzard/decision-model-benchmark README). The jev-benchmarks pilot reports AG News 0.910 versus 0.700 for the GLiNER2.5 encoder, Banking77/BTZSC 0.870 versus 0.610, a DAIR Emotion accuracy tie with Jev substantially worse calibrated (Brier 0.846 versus 0.668, negative log-likelihood (NLL) 5.588 versus 1.381, zero probability on the true label for 16% of examples), and GLiNER at about 44ms p50 on an Apple M4 Max central processing unit (CPU) versus Jev at 236 to 256ms hosted, with Jev slightly faster on the 72-label condition (246ms versus 296ms) (AbdelStark/jev-benchmarks README). These show large constrained decoders can be accurate and calibrated at high latency and cost, but say nothing direct about sub-1B decoders, whose zero-shot quality the cards show to be much weaker.

Analysis

Latency: the constraint tax is dead, the token loop is not. XGrammar’s reported near-zero end-to-end overhead removes masking cost from the comparison. What remains is structural: a choice over single-token labels costs a small decoder about one forward pass, while JSON output and multi-token labels cost one pass per token, and long input states impose prefill cost on both architectures. For short states and single-token labels, a sub-1B decoder under guided choice is plausibly latency-competitive with a same-size encoder. For structured output, the decoder pays per token while a native encoder head pays once, which preserves the parent survey’s latency argument exactly where output is longest. The missing quantity is a measured per-decision latency for a named sub-1B decoder under guided choice on fixed hardware.

Accuracy at matched size favors encoders, and small-decoder zero-shot numbers do not rescue the decoder side. Ettin’s 400M-encoder-beats-1B-decoder result on MNLI is direct architectural evidence at the relevant sizes, and the SmolLM2 cards show zero-shot decision-adjacent scores (MMLU cloze in the low 30s, IFEval under 45) far below fine-tuned classifier performance. Constrained decoding cannot close this gap in principle: it restricts output to valid options but contributes no information about which option is correct. The fair reply, that fine-tuning plus constrained decoding is the relevant configuration, is currently evidence-free: no located source reports that result. Until such a number exists, the accuracy presumption at matched size belongs to encoders.

Calibration is the weakest point of the decoder case and the least measured. A decoder’s label score under guided choice is a renormalized slice of next-token probabilities, possibly a product over multiple tokens for multi-token labels, while an encoder classifier emits a softmax that can be temperature-fitted as a label belief (the Guo et al. recipe, arXiv:1706.04599). Whether single-parameter temperature scaling transfers cleanly to guided-choice decoder scores at sub-1B size is UNVERIFIED: Laya’s ECE 0.081 result is an encoder result, and the benchmark’s strong LLM calibration numbers (ECE 0.039 to 0.122) come from large models whose calibration need not transfer downward. The multi-token-label problem the parent raised (shared prefixes, length effects) also survives constrained decoding untouched, since the constraint selects allowed tokens without normalizing their probabilities into a label belief. Any small-decoder decision proposal needs a stated calibration story with measured ECE, not an assumed one.

Flexibility is the genuine decoder advantage. Three items favor small decoders. Dynamic label sets are native to prompting: new labels are text in the prompt, while Laya-style encoder heads must fit every option into a fixed token budget (the documented cause of its Banking77 weakness at 0.425). Multilingual coverage comes with the base model: Qwen3 reports 119 languages and dialects, while Laya maintains separate English and multilingual checkpoints behind a script-detecting router. Structured JSON output is routine for a decoder with a grammar (GBNF forces valid JSON; vLLM and XGrammar serve it at near-zero overhead), while the parent survey lists typed JSON output as an open problem with no first-class non-autoregressive primitive. These advantages matter most when labels, languages, or schemas vary faster than an encoder pipeline can be re-fitted; they say nothing about accuracy or calibration on any fixed task.

Operations favor small decoders on deployability and encoders on serving efficiency at scale. A 135M to 600M decoder with weights on Hugging Face runs locally under llama.cpp or a small serving stack, with no API dependency and no per-decision fee, against Jev’s measured $0.07 per 1,000 decisions and 255-choice cap. Against Laya the comparison is closer: both are local and small, and Laya’s measured 7.2ms per batched question on T4 is the number a decoder deployment must beat. Throughput per watt at high volume plausibly favors the encoder (one pass per decision regardless of output length), while single-artifact simplicity favors the decoder when the same model must also handle generation alongside decisions. Both halves of that sentence need measurements.

Implications for Jev, Laya, and Maestro Flash

For Jev, the small-decoder analysis sets up a testable open alternative rather than a verdict. Jev’s measured profile (mid-pack Banking77 accuracy, flat 264 to 276ms latency across option counts, ECE 0.246, 13% flip rate, hard 255-choice cap) is consistent with several architectures, and its actual architecture remains UNVERIFIED. The flat latency curve is sometimes read as evidence of a single-pass architecture, but flatness is also what batched or cached serving produces, so that inference is weak. What this paper adds is a concrete challenger: a named sub-1B instruct model under guided choice, evaluated on the same suites (Banking77 77-way, spam, permuted options, forced-uncertainty items) with the same metrics (accuracy, ECE, flip rate, p50 latency, cost). If that configuration matches Jev’s accuracy at lower cost with better calibration, Jev’s moat narrows to API convenience; if it falls far short, that shortfall quantifies what Jev’s closed training buys.

For Laya, the analysis sharpens rather than threatens the encoder choice. At 421M and 322M parameters Laya sits inside the sub-1B band, so Ettin’s matched-size encoder advantage applies directly: swapping its backbone for a same-size decoder would be expected to lose classification accuracy. The decoder side contributes two constructive suggestions. First, Laya’s high-cardinality weakness (options competing for a fixed head token budget, Banking77 at 0.425) is a problem decoders do not have, since options as generable text face no shared budget; a hybrid that keeps the encoder head for small option sets and routes large ones elsewhere would address the weakness without abandoning the architecture. Second, Laya has no JSON primitive while grammar-constrained small decoders emit valid JSON routinely; if typed JSON enters Laya’s scope, a small decoder sidecar for that primitive is a smaller change than inventing a non-autoregressive JSON head. Laya’s per-bucket temperature calibration would not transfer to such a sidecar without new measurement.

For Maestro Flash, Velofy’s proposed decision model, the implications are decision criteria and a measurement plan, not results, since Maestro Flash has not been trained and nothing is claimed about its performance. On backbone choice, the evidence supports an encoder default with a decoder comparison arm: Ettin favors encoders at matched size, Laya demonstrates the encoder path end to end with measured calibration, and the small-decoder alternative is a hypothesis with weak zero-shot priors and no located fine-tuned decision numbers. The comparison arm should be a named sub-1B instruct decoder (SmolLM2-360M-Instruct and Qwen3-0.6B are the natural first candidates given public weights and cards) under guided choice, evaluated on the same typed-decision suite as the encoder candidate with accuracy, ECE, flip rate, p50 and 95th-percentile (p95) latency, and cost per 1,000 decisions all reported. The suite must include the conditions where decoders are hypothesized to win (many options, multilingual input, JSON output), not only conditions where encoders are known to win, or the comparison proves nothing. The decision rule can be stated in advance: adopt the decoder backbone only if it matches the encoder on accuracy and ECE while winning elsewhere, since accuracy and calibration are the load-bearing requirements for confidence-gated automation.

Limitations

No new experiments were run; every number is someone else’s measurement, attributed inline. The central head-to-head study (named sub-1B decoder with constrained decoding versus named encoder on a shared typed-decision suite) was not located and may not exist publicly, so the core comparison is assembled from adjacent evidence with an explicit UNVERIFIED cell. Third-party benchmark numbers are taken from project READMEs as read, not reproduced. Model-card numbers are vendor self-reports on generative benchmarks, useful as priors but not as decision-task evidence. Full papers were not re-analyzed beyond abstracts and fetched documentation. Multilingual aspects are noted only where a fetched source states them (Qwen3’s 119 languages and dialects); no dedicated small-decoder Indic decision evaluation was located. The field moves quickly, so numbers here are dated observations, not constants.

Open questions

1. What does a named sub-1B instruct decoder score, under guided choice on fixed hardware, against a matched-size encoder on a shared typed-decision suite measuring accuracy, ECE, flip rate, and p50 latency?
2. Does single-parameter temperature scaling calibrate guided-choice decoder scores at sub-1B size, and how does the resulting ECE compare with encoder calibration on the same suite?
3. On the conditions hypothesized to favor decoders (many options, multilingual input, JSON output), does a small decoder actually beat an encoder head, and at what output length does the token-loop latency cost erase the advantage?

References

1. SmolLM2, arXiv:2502.02737, https://arxiv.org/abs/2502.02737
2. Qwen3 Technical Report, arXiv:2505.09388, https://arxiv.org/abs/2505.09388
3. Gemma 3 Technical Report, arXiv:2503.19786, https://arxiv.org/abs/2503.19786
4. ModernBERT, arXiv:2412.13663, https://arxiv.org/abs/2412.13663
5. Seq vs Seq (Ettin paired encoders and decoders), arXiv:2507.11412, https://arxiv.org/abs/2507.11412
6. Efficient Guided Generation for Large Language Models (Outlines), arXiv:2307.09702, https://arxiv.org/abs/2307.09702
7. XGrammar, arXiv:2411.15100, https://arxiv.org/abs/2411.15100
8. Fast Structured Decoding for Sequence Models, arXiv:1910.11555, https://arxiv.org/abs/1910.11555
9. GLiNER, arXiv:2311.08526, https://arxiv.org/abs/2311.08526
10. GLiClass, arXiv:2508.07662, https://arxiv.org/abs/2508.07662
11. On Calibration of Modern Neural Networks (Guo et al.), arXiv:1706.04599, https://arxiv.org/abs/1706.04599
12. Outlines README (dottxt-ai/outlines), https://github.com/dottxt-ai/outlines
13. XGrammar README (mlc-ai/xgrammar), https://github.com/mlc-ai/xgrammar
14. llama.cpp GBNF grammars guide, https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md
15. Guidance README (guidance-ai/guidance), https://github.com/guidance-ai/guidance
16. vLLM structured outputs documentation, https://docs.vllm.ai/en/latest/features/structured_outputs.html
17. SmolLM2-135M model card (HuggingFaceTB/SmolLM2-135M), https://huggingface.co/HuggingFaceTB/SmolLM2-135M
18. SmolLM2-360M model card (HuggingFaceTB/SmolLM2-360M), https://huggingface.co/HuggingFaceTB/SmolLM2-360M
19. Qwen3-0.6B model card (Qwen/Qwen3-0.6B), https://huggingface.co/Qwen/Qwen3-0.6B
20. gemma-3-270m model page (google/gemma-3-270m), https://huggingface.co/google/gemma-3-270m
21. Laya README (NandhaKishorM/laya), https://github.com/NandhaKishorM/laya
22. Decision-model benchmark (nibzard/decision-model-benchmark), https://github.com/nibzard/decision-model-benchmark
23. jev-benchmarks pilot (AbdelStark/jev-benchmarks), https://github.com/AbdelStark/jev-benchmarks

← All papers