Typed Decisions Without Generation: A Survey

Status: survey and position paper. This paper synthesizes prior published work and two publicly documented systems: Laya, an open project, and TypeSafe Jev, a closed application programming interface (API) known only through third-party benchmarks. It reports no new experiments and no Maestro Flash results, because Maestro Flash has not been trained. All numeric claims below are attributed to their original source.

Velofy research report, revision 3 (r3), 2026-09-23.

Abstract

We survey the emerging category of models that answer closed, typed questions about an input without autoregressive text generation: pick one of k labels, produce a bounded score, output a calibrated probability, extract a span, or fill a typed schema. We call this category typed decisions. It sits between open-ended generation on one side and classic supervised classification on the other. Typed decisions are posed as explicit, schema-typed questions over arbitrary input state at inference time, often with a dynamically specified option set, rather than trained against one fixed label space. We trace the category’s technical roots through non-autoregressive sequence generation, encoder-only pretraining, calibration theory, and reinforcement learning (RL) against proper scoring rules. We then examine two concrete systems that instantiate it end to end: the open project Laya and the closed system TypeSafe Jev, surveyed only through third-party benchmarks because no primary technical documentation was located on arXiv. Finally we identify open problems that motivate a concrete design proposal, Maestro Flash, developed as a companion technical report in this graph: high-cardinality option sets, ordinal scoring, structured JavaScript Object Notation (JSON) output, multilingual and especially Indic-language calibration, and option-order robustness.

1. Introduction

Large language models (LLMs) are usually evaluated and marketed on open-ended generation quality: writing, coding, multi-step reasoning. But a large and often invisible share of production LLM usage is not open-ended at all. Consider a customer support platform that must route every incoming ticket to one of six departments, assign an urgency score from a four-point rubric, decide whether the customer is threatening to churn, and decide whether a refund was explicitly requested, for every single ticket, at whatever volume the business receives. Or consider a document-processing pipeline that must decide, for every incoming invoice, whether it is a duplicate, extract the invoice number and due date, and classify the vendor into one of a few hundred categories. None of this requires the model to write anything new. It requires the model to answer closed, typed questions, reliably, cheaply, and with an honest notion of how confident it is.

We use the term typed decision for this class of problem, and we define it by four properties that distinguish it from both classic supervised classification and open-ended generation:

1. Closed output space. The set of valid answers is small and enumerable at inference time (a label set, a bounded numeric rubric, a boolean, or a typed schema), not “any string.”
2. Dynamic question specification. Unlike classic classification, the label set or schema is often specified at inference time, not baked into the model’s output layer at training time. A support-routing model must handle whatever department taxonomy today’s customer configured, not a fixed taxonomy learned once.
3. Arbitrary input state. The input is not a fixed-format record; it is free text, structured documents, or a mix, in potentially many languages.
4. Calibration as a first-class requirement. Typed decisions are usually wired into automated pipelines with a confidence-gated escalation path (for example, “route automatically above 0.85 confidence, else escalate to a human”). A well-calibrated probability is therefore not a nice-to-have evaluation metric. It is the mechanism by which the system decides whether to trust its own output.

This survey traces the technical lineage of systems built for this category, examines the two concrete systems that currently instantiate it, and identifies the open problems that motivate further work. Those problems are taken up in a companion design proposal, the Maestro Flash technical report. This survey does not evaluate Maestro Flash, because it does not yet exist as a trained model.

2. Why not just prompt a decoder?

The obvious way to answer a typed question with an LLM is to prompt an autoregressive decoder (Qwen3, arXiv:2505.09388; Gemma 3, arXiv:2503.19786; Phi-4, arXiv:2412.08905; SmolLM2, arXiv:2502.02737) to emit the label or JSON object as text, then parse the result. This works, and is the default approach in most production systems today. It has three specific costs that make the typed-decision category a distinct research problem rather than a solved special case of generation:

Latency. An autoregressive decoder pays one forward pass per generated token. Even a single-word label costs at least one full forward pass through the entire decoder stack, and a JSON object with several fields costs one pass per token of that object. A single-pass head pays one forward pass however many options or fields it scores. Published numbers show the size of the gap in practice, with a caveat. Laya’s README reports that TypeSafe Jev, called as a hosted API, has been measured by two separate third-party benchmarking projects (github.com/AbdelStark/jev-benchmarks and github.com/nibzard/decision-model-benchmark) at 236–276 ms median (p50) latency for a single typed decision. Laya itself, using a single non-autoregressive forward pass, reports 32.8–39.5 ms p50 on a T4 graphics processing unit (GPU) for the same style of question, roughly 6 to 8 times faster by those figures. The comparison is not like for like: Jev’s figures include a network round trip to a hosted service, and Jev’s architecture is not public, so the gap cannot be assigned to the decode loop alone. The architectural argument rests on the pass count, not on this pair of numbers.

Calibration and multi-token labels. A decoder’s output “probability” for a generated label is the product of per-token conditional probabilities along the sampled path. This is a well-defined quantity, but it is not the same quantity as “the model’s calibrated belief that this is the correct label” in the sense a classification softmax provides, and it becomes awkward when labels share prefixes or differ in token length. Confidence failures are costly whatever the architecture. One independent study (AbdelStark/jev-benchmarks, cited in Laya’s README) found Jev assigning zero probability to the true label on 16% of DAIR Emotion examples, a hard failure mode for any pipeline that branches on confidence.

Parseability. Generated text must be parsed back into a typed value. This is usually fine for a single clean label, but grows fragile for structured JSON output with nested fields, especially under adversarial or malformed input, or when the schema itself is unusual. Grammar-constrained decoding partially addresses this but does not remove the basic mismatch between “generate then parse” and “answer directly.”

None of this is a claim that decoders are unsuitable for typed decisions in general. A sufficiently large, well-calibrated decoder with constrained decoding can do this task adequately, and remains the right choice when the same model must also handle open-ended generation in the same pipeline. The claim is narrower. For the specific, common case where a system needs only typed decisions, at volume, within a latency and cost budget, a purpose-built non-generative architecture is a better fit, and that architecture is now a distinct object of study rather than a decoder workaround.

3. Technical lineage

3.1 Non-autoregressive generation

The idea of removing the autoregressive factorization from sequence generation predates the typed-decision framing by nearly a decade, originating in machine translation. Gu et al. (arXiv:1711.02281) introduced a Transformer decoder that generates all output tokens in parallel. It is trained by knowledge distillation from an autoregressive teacher, plus a fertility latent variable to handle the multimodality problem (multiple valid translations of one source sentence), at a reported cost of roughly 2.0 BLEU (Bilingual Evaluation Understudy) points relative to the autoregressive teacher for an order-of-magnitude inference speedup. This is the direct ancestor of every “one forward pass, no decode loop” system that follows, including Laya and the Maestro Flash proposal.

arXiv:2204.09269 surveys the following decade of non-autoregressive generation research. Notably, it extends the frame beyond machine translation to automatic speech recognition, controllable generation, question answering, summarization, grammatical error correction, and other structured tasks. That established, years before the typed-decision framing became commercially relevant, that “generate everything in parallel” is a general technique, not a translation-specific trick. The most current synthesis, arXiv:2508.08712 (2025), connects this classical non-autoregressive line to the newer parallel decoding of diffusion language models (§3.4 below). That matters because it suggests two independently arrived-at technical lineages, classification-style non-autoregressive heads and diffusion-style iterative parallel refinement, are converging on the same underlying goal.

3.2 Encoder-only pretraining

Typed decisions are, at the representation-learning level, classification and regression problems, and the strongest classification backbones remain encoder-only bidirectional models. The 2024–2025 period saw a substantial revival of this architecture family after several years of decoder-only dominance in the broader field. ModernBERT (arXiv:2412.13663) modernized the Bidirectional Encoder Representations from Transformers (BERT) recipe (rotary embeddings, GeGLU, alternating local and global attention, Flash Attention, native 8192-token context, 2T training tokens) and set a new open Pareto frontier for English encoders; it is Laya’s own English backbone. mmBERT (arXiv:2509.06888) extended the same recipe to 1,800+ languages with an explicit low-resource-language annealing phase late in training, and is Laya’s multilingual backbone.

The most methodologically important recent contribution to this specific question (is an encoder actually better than a decoder for classification-shaped tasks, controlling for everything else?) is Ettin (arXiv:2507.11412). It trains paired encoder-only and decoder-only models on identical data, architecture, and recipe across a range of sizes (17M to 1B parameters). The paper reports that encoders win decisively on classification and retrieval at matched size, and decoders win on generative tasks. Critically for the typed-decision framing, it also reports that continuing to train a decoder into an encoder-shaped task underperforms training an encoder natively from the start. This is the strongest available evidence, independent of any typed-decision-specific system, that the architectural choice both Laya and Maestro Flash make (encoder-native, not decoder-repurposed) is well-founded.

3.3 Calibration

A typed decision’s usefulness in an automated pipeline depends on whether its stated confidence is trustworthy, not just on whether its argmax is usually correct. Guo et al. (arXiv:1706.04599) established in 2017 that modern (post-2015) deep networks are systematically overconfident relative to older, shallower architectures, despite higher accuracy, and that single-parameter temperature scaling is a surprisingly effective and cheap remedy. This finding, made on image classifiers, replicates closely on encoder-based typed-decision models nearly a decade later. Laya’s own BENCHMARKS.md reports that both of its checkpoints ship overconfident, with expected calibration error (ECE) of 0.466 and 0.314. A single fitted temperature per (question type, option count) brings ECE down to 0.081 and 0.106 respectively. That is a direct, measured replication of Guo et al.’s core result on this exact model class.

3.4 Diffusion language models

A separate line of work replaces autoregressive decoding with iterative denoising over discrete tokens. SEDD (arXiv:2310.16834) introduced score entropy, a discrete analogue of score matching, and reported perplexity reductions of 25–75% over prior diffusion language model approaches at matched size. LLaDA (arXiv:2502.09992), an 8B masked-diffusion model trained from scratch, is reported to rival LLaMA3-8B on general benchmarks. That is evidence that non-autoregressive generation can scale to competitive quality, not merely serve as a latency optimization for small models. Dream 7B (arXiv:2508.15487) initializes from an autoregressive checkpoint and is reported to match LLaMA3-8B and Qwen2.5-7B while also supporting arbitrary-order generation and infilling. These systems are not typed-decision systems as such; they remain general-purpose language models. But they show that the boundary between “generation” and “parallel structured decision” is less sharp than the typed-decision framing might suggest, and they are a plausible future mechanism for the span/json structured-output primitives that this survey’s companion technical report proposes.

3.5 Proper scoring rules and RL

The final piece of the lineage is the training objective. A strictly proper scoring rule is a loss function for probabilistic forecasts that is uniquely minimized in expectation by reporting one’s true belief. The logarithmic score (cross-entropy) is one member of this family, but not the only one, and not necessarily the best-behaved one for a given downstream use. Shao et al. (arXiv:2405.18906) show, at decoder scale up to 13B parameters, that substituting a non-local strictly proper scoring rule (Brier or spherical score) for the logarithmic score as the training loss, with no other hyperparameter changes, yields substantial generation-quality improvements. RLCR (arXiv:2507.16806) applies a related idea inside an RL loop: add a Brier-score term to the reward so a model is explicitly penalized for the gap between its stated confidence and its realized correctness. It reports improved calibration with no accuracy cost. Group Relative Policy Optimization (GRPO, arXiv:2402.03300) supplies the RL algorithm skeleton most current systems in this space use: a group-relative baseline in place of a learned critic, which substantially reduces RL compute overhead. Laya’s README names this exact combination, RL against proper scoring rules, as its training method and calls it “RLCD”. No separate technical paper documenting Laya’s implementation of this method was located on arXiv; Laya’s public README and BENCHMARKS.md are the only primary sources available for this review.

4. Two systems

TypeSafe Jev. A closed, hosted API. No architecture or training paper for Jev was located on arXiv as of this review. Every number attributed to Jev in this survey is drawn from third-party benchmarking projects (github.com/AbdelStark/jev-benchmarks, github.com/nibzard/decision-model- benchmark), as cited in Laya’s own README and BENCHMARKS.md. These numbers were measured by parties without access to Jev’s internals, under conditions (sample size, prompt format) that may differ from any controlled comparison. Reported figures include: AG News 0.910, Banking77 0.870 (on a 72-label condition in the AbdelStark pilot), DAIR Emotion 0.480 (with Brier 0.846 and 16% zero-probability-on-true-label failures), ECE 0.246, Banking77 option-order flip rate 0.13, and p50 latency 236–276 ms.

Laya (github.com/NandhaKishorM/laya, Apache 2.0). An open, encoder-based typed-decision engine that answers choice/score/noul questions in one forward pass, with ModernBERT-large for English and mmBERT-base for 100+ other languages, dispatched by a script- and language-detecting Router. It is trained with RL against proper scoring rules (“RLCD”) followed by post-hoc temperature calibration. Laya’s own measured numbers, from its README and BENCHMARKS.md read directly for this survey, include:

• 0.766 accuracy on its own 2,000-decision typed-decisions benchmark after fine-tuning, against 0.727 for Jev’s published figure on the same style of benchmark (though not the identical dataset or measurement conditions).
• 32.8 ms p50 latency on a T4, and ECE 0.081 after calibration.
• A documented, load-bearing limitation: near-chance performance on its own benchmark zero-shot (0.362 and 0.342 for its two base checkpoints, both below a 0.461 majority-class baseline). In other words, essentially all of its reported capability comes from task-specific fine-tuning rather than from the base checkpoints alone.

Laya’s clearest documented weaknesses, stated in its own README, are:

• High-cardinality choice questions (Banking77: 0.425–0.492 against Jev’s published 0.870), which Laya’s authors attribute to a token-budget architectural constraint, not a capability ceiling.
• Soft-distribution matching against a teacher (0.471 against Jev’s published 0.580).
• Ordinal score questions generally (SST-5, the five-class Stanford Sentiment Treebank: 0.372).
• Confident failure on non-Latin scripts when the wrong checkpoint is selected (Khmer: 0.000 accuracy at 0.952 confidence), addressed by routing rather than by within-model calibration.

5. Open problems

Drawing on Laya’s own documented limitations and the broader literature surveyed above, we identify five open problems for the typed-decision category as a whole:

1. High-cardinality option sets. Every system surveyed here degrades on large label spaces when option representations compete for a shared, fixed token budget. This is a solvable architectural problem (decouple option encoding from the state’s context budget), but no published system in this survey has reported a fix, only a workaround: Laya’s predict_shortlist embedding-based shortlisting, which Laya’s own documentation notes has not been remeasured by Laya’s own team, only by a third-party issue reporter.
2. Ordinal scoring. Every system surveyed treats, or has been documented to treat, rubric-based scoring as unordered classification rather than as ordinal regression, and it is consistently the weakest primitive where measured (Laya’s SST-5: 0.372).
3. Structured JSON output. No system surveyed here exposes typed JSON-schema output as a first-class non-autoregressive primitive. This remains an open architectural problem, despite the general non-autoregressive structured-output framing existing in the literature since at least arXiv:2204.09269.
4. Multilingual calibration, and Indic languages specifically. Laya’s documented solution is to route between separately trained checkpoints based on script and language detection before the forward pass. It works, but leaves open whether a single backbone with finer-grained calibration can achieve comparable results without the operational cost of keeping several checkpoints resident. Laya’s own per-language data shows every Indic language it benchmarks sitting well below its English number even on the multilingual checkpoint. This review located no dedicated, human-verified typed-decision benchmark for Indic languages, as opposed to machine-translated data or professional translations made only for natural language inference.
5. Option-order robustness. Both systems surveyed here show non-trivial sensitivity to the order in which options are presented (Laya’s own measured flip rates of 0.15-0.23 at 20 options against Jev’s published 0.13). This suggests current training recipes do not randomize option order enough during training.

Each of these open problems is addressed as a specific design proposal in the companion Maestro Flash architecture and roadmap and technical report. Problems 2 and 4 are also the subjects of dedicated companion papers: the calibration proposal and the Indic-first benchmark proposal respectively.

6. Limitations of this survey

This survey covers two systems in the typed-decision category as narrowly defined here (§1). It does not attempt exhaustive coverage of the broader structured-prediction or information-extraction literature, which is far larger and substantially predates the LLM era. TypeSafe Jev is surveyed only through third-party benchmarks and Laya’s own citations of them; we could not verify these independently, and Jev’s own architecture, training method, and official benchmark methodology remain unknown to this review. Laya is surveyed through its own public documentation, read directly rather than through secondary sources, but its numbers are Laya’s own self-reported measurements, not independently reproduced by this project.

References

1. Gu, Bradbury, Xiong, Li, Socher. “Non-Autoregressive Neural Machine Translation.” arXiv:1711.02281
2. “A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond.” arXiv:2204.09269
3. “A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models.” arXiv:2508.08712
4. Warner et al. “Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder…” arXiv:2412.13663
5. Marone, Weller, Fleshman, Yang, Lawrie, Van Durme. “mmBERT: A Modern Multilingual Encoder with Annealed Language Learning.” arXiv:2509.06888
6. Weller, Ricci, Marone, Chaffin, Lawrie, Van Durme. “Seq vs Seq: An Open Suite of Paired Encoders and Decoders.” arXiv:2507.11412
7. Guo, Pleiss, Sun, Weinberger. “On Calibration of Modern Neural Networks.” arXiv:1706.04599
8. Lou, Meng, Ermon. “Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution.” arXiv:2310.16834
9. Nie et al. “Large Language Diffusion Models.” arXiv:2502.09992
10. “Dream 7B: Diffusion Large Language Models.” arXiv:2508.15487
11. Shao, Meng, Liu, Zhou. “Language Generation with Strictly Proper Scoring Rules.” arXiv:2405.18906
12. “Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty.” arXiv:2507.16806
13. “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.” arXiv:2402.03300
14. Qwen Team. “Qwen3 Technical Report.” arXiv:2505.09388
15. Gemma Team, Google DeepMind. “Gemma 3 Technical Report.” arXiv:2503.19786
16. “Phi-4 Technical Report.” arXiv:2412.08905
17. “SmolLM2: When Smol Goes Big – Data-Centric Training of a Small Language Model.” arXiv:2502.02737
18. Laya (github.com/NandhaKishorM/laya), README.md and BENCHMARKS.md, read directly, 2026. Third-party Jev benchmarks cited therein: github.com/AbdelStark/jev-benchmarks, github.com/nibzard/decision-model-benchmark.

← All papers