SearchBook a call

RESEARCH / PAPERS

proposal

Maestro Flash: A Non-Autoregressive Multilingual Model for Typed Decisions and Structured Outputs

Proposal. This describes planned work; any numbers marked TARGET are goals, not measured results.

Design proposal for Maestro Flash, an untrained multilingual encoder for typed decisions: architecture, training recipe, evaluation plan and compute.

Velofy Lab ·

Status: proposal and technical report with planned experiments. Maestro Flash has not been trained. Every number in this paper is either (a) a cited result from a published paper or from Laya’s own published README or BENCHMARKS.md, attributed as such, or (b) labelled TARGET. No number in this paper is a Maestro Flash measurement.

Velofy research report, revision 3 (r3), 2026-09-23.

Abstract

A growing share of production language-model traffic is not open-ended generation. It is a closed decision over a fixed set of typed outputs: pick one of N labels, produce a bounded score, extract a span, or fill a typed JavaScript Object Notation (JSON) schema. Autoregressive decoders answer such questions by generating text token by token and parsing the result. That is slow, occasionally unparseable, and gives no first-class calibrated probability for a multi-token label. Two recent systems answer such questions without generating text: the closed application programming interface (API) “TypeSafe Jev”, whose architecture is not public, and the open, Apache-2.0 project Laya (github.com/NandhaKishorM/laya), which uses a single forward pass through an encoder trained with reinforcement learning (RL) against strictly proper scoring rules. We propose Maestro Flash, Velofy’s own open small model for this category, positioned against both. Maestro Flash’s design differs from Laya in three concrete ways: (1) it adds span and schema-typed json output primitives to Laya’s choice/score/noul set; (2) it targets a single unified multilingual backbone with per-(question type, option count, language) calibration in place of Laya’s documented two-checkpoint-plus-router architecture, motivated by Laya’s own reported script-collapse failure mode; (3) it treats Indic-language typed-decision accuracy as a first-class, human-verified evaluation target rather than one slice of a 51-language average. This paper is a technical report of the design: architecture, training recipe (supervised multi-task fine-tuning, teacher distillation, RL against proper scoring rules, calibration), evaluation plan, and compute budget for both an 8×H100-week training run and deployment and fine-tuning on a single RTX graphics processing unit (GPU). We report zero trained-model results. We report the literature and prior published numbers (Laya’s, Jev’s third-party-measured numbers, and the academic papers each design choice draws on) that motivate each design decision. We also specify, in advance, the metrics and go/no-go comparisons that will decide whether Maestro Flash is worth releasing.

First principles

The design rests on a few facts about typed decisions and encoders. Each is either basic arithmetic or comes from a source cited later in this paper.

  1. The answer space is known before the input arrives. For choice, score and noul, the model does not need to write anything. It needs to put a probability on each allowed answer. One forward pass that scores every option does that, and leaves nothing to parse.
  2. A shared token budget is divided among the options. If the state, the question and all k options share one fixed window, each option gets roughly (budget minus overhead) / k tokens. Laya’s README works this out as (256 - 16) // 77 ≈ 3–4 tokens per label at 77 options. Option count and per-option detail then trade off directly. Encoding options in a separate pass removes that trade-off (§3.2).
  3. Ordered labels carry information that a plain softmax discards. A rubric’s levels have an order. A softmax over unordered classes treats “1 versus 2” the same as “1 versus 5”. A cumulative-link head keeps the order (§3.3).
  4. A flat schema is a set of independent typed fields. If every leaf is an enum, a bounded number, a boolean or a span, the existing heads can predict all of them in parallel and a deterministic step can assemble the object. Nested and recursive schemas break that independence, so they are out of scope for v1 (§3.4).
  5. Calibration is a fitted mapping, and each mapping needs held-out data. One temperature per (question type, option count, language) cell multiplies the number of cells. More cells means fewer held-out items per cell, which bites hardest for low-resource languages (§4, §7).
  6. Calibration can make a model honestly unsure, but it cannot make it accurate. Laya’s English checkpoint answers Khmer at 0.000 accuracy with 0.952 confidence. A per-language temperature could lower that confidence. Accuracy on scripts the backbone does not represent has to come from the backbone and the training data.

1. Introduction

The dominant framing of language model capability is generative: given a prompt, produce fluent, open-ended text. A large and growing fraction of real deployed use, however, is not generative in this sense. It is a closed decision. A support ticket needs a department label from a fixed set. A ticket needs an urgency score from a bounded rubric. An email needs a yes/no judgment on whether it requests a refund. An invoice needs three fields extracted into a typed JSON object. In each case the output space is small, known in advance, and structurally simple. What is hard is getting the probability right, not generating novel text.

Using an autoregressive decoder for this class of problem is a mismatch on several axes. First, latency: a decoder must run one forward pass per output token, even when the “output” is a single word from a five-word label set. Second, calibration: cross-entropy training on next-token prediction optimizes token-level likelihood, not decision-level calibration, and a label that spans multiple sub-word tokens has no clean, single probability without an ad hoc aggregation rule. Third, parseability: generated text must be parsed back into a typed value, and parsing can fail or silently drift from the intended schema.

Two systems have converged on the same alternative: answer the typed question directly, with no generation and no parsing step. TypeSafe Jev is a closed, hosted product. No paper describing its architecture or training was located on arXiv as of this review, so its internals are unknown. Every Jev number cited in this paper is third-party-published (via github.com/AbdelStark/jev-benchmarks and github.com/nibzard/decision-model-benchmark) rather than independently reproduced, and should be read with that caveat. Laya (github.com/NandhaKishorM/laya, Apache 2.0) is the open reference implementation of the idea: a classification and regression head over an encoder, run in one forward pass. It answers choice (pick one of k options), score (an ordinal value on a rubric), and noul (a calibrated boolean probability) questions. It uses ModernBERT-large (arXiv:2412.13663) for English and mmBERT-base (arXiv:2509.06888) for 100+ other languages, dispatched by a router, and is trained with what its README calls “RLCD”: reinforcement learning against strictly proper scoring rules.

This paper proposes Maestro Flash, Velofy’s own model in this category, and reports it as what it currently is: a design. It is informed by a systematic literature review (published in this graph as the companion survey of typed decisions without generation) and by a close reading of Laya’s own published README, BENCHMARKS.md, and research/README.md, which we read directly rather than summarizing secondhand. We make three concrete design departures from Laya, each motivated by a specific, citable finding in Laya’s own documentation or in the broader literature:

  1. New primitives. Laya’s own limits section states plainly that it exposes choice/score/ noul only; extraction and structured-JSON output are not first-class primitives. We add span (extractive) and a schema-typed json head (§3.4).
  2. Unified backbone. Laya’s own BENCHMARKS.md documents that its English checkpoint achieves 0.000 accuracy at 0.952 mean confidence on Khmer, and that macro expected calibration error (ECE) across 51 languages for the English checkpoint is 0.733. The model is confidently wrong on scripts it cannot read. Laya’s own stated fix is to route to a different checkpoint before the forward pass, using sub-millisecond script detection, rather than to fix the single model’s calibration. We propose to first test whether a single mmBERT-backbone model with finer-grained (per-language) calibration can substantially close this gap without a router. mmBERT’s own multilingual macro-ECE (0.387, per Laya’s own multilingual-checkpoint numbers in BENCHMARKS.md) is already far better than the English checkpoint’s. We fall back to routing only if this does not hold up under evaluation.
  3. Indic-first evaluation. Laya’s per-language table in BENCHMARKS.md shows that every Indic language it benchmarks (hi, bn, ta, te, ml, kn, ur) sits well below its English number even on the multilingual checkpoint (for example, Hindi 0.430 against English 0.680 on MASSIVE intent, macro over both checkpoints). We treat this not as an incidental data point but as the evaluation target. Maestro Flash proposes a dedicated, human-verified Indic typed-decision benchmark (the companion Indic-first benchmark proposal) rather than reporting Indic performance only as a few rows inside a 51-language table.

The remainder of this paper is organized as follows. §2 reviews related work. §3 describes the proposed architecture. §4 describes the training recipe. §5 describes the evaluation plan. §6 gives the compute budget. §7 discusses limitations. §8 concludes.

Encoder backbones. ModernBERT (arXiv:2412.13663) modernized the Bidirectional Encoder Representations from Transformers (BERT) recipe (rotary position embeddings (RoPE), GeGLU gated activations, alternating local and global attention, native 8192-token context) and is Laya’s English backbone. mmBERT (arXiv:2509.06888) extends this recipe to 1,800+ languages on 3T tokens with an annealed low-resource-language decay phase, and is Laya’s multilingual backbone; we adopt it as Maestro Flash’s primary backbone candidate (§3.1). Ettin (arXiv:2507.11412) provides the methodologically cleanest evidence that encoder-only architectures beat matched-size, matched-data, matched-recipe decoders on classification tasks, and that adapting a decoder into an encoder underperforms training an encoder natively. That is direct support for the encoder-first design Laya and Maestro Flash share. DeBERTaV3 (arXiv:2111.09543) and MuRIL (arXiv:2103.10730) are the relevant prior-generation and Indic-specific encoder precedents, respectively. MuRIL’s transliteration-augmentation training signal is adopted directly into Maestro Flash’s Indic data pipeline (§3.5).

Non-autoregressive generation. The mechanism both Laya and Maestro Flash rely on, parallel typed output in a single forward pass, descends from non-autoregressive machine translation (arXiv:1711.02281). It is surveyed comprehensively in arXiv:2204.09269 and, more recently, alongside parallel decoding in diffusion language models in arXiv:2508.08712. We treat Maestro Flash’s json-typed head (§3.4) as an instance of this general framing: structured output as a set of parallel typed decisions dispatched by schema, not as sequential token generation.

Diffusion language models. SEDD (arXiv:2310.16834), LLaDA (arXiv:2502.09992), and Dream 7B (arXiv:2508.15487) demonstrate that diffusion-style parallel decoding can match autoregressive quality at 7-8B scale. They are cited as a longer-horizon alternative decoder-side mechanism for the span/json primitives. At 7-8B they exceed this project’s single-RTX-GPU deployment budget (§6) and are not part of the v1 design.

Calibration and proper scoring rules. Guo et al. (arXiv:1706.04599) established that modern neural networks are systematically overconfident and that temperature scaling is an effective, cheap fix. Laya’s own README directly replicates this finding on an encoder-based typed-decision model: ECE falls from 0.466 to 0.081 after temperature refitting. Shao et al. (arXiv:2405.18906) show that training language generation directly against strictly proper scoring rules (Brier, spherical), rather than the standard logarithmic score, yields quality improvements at up to 13B decoder scale. RLCR (arXiv:2507.16806) adds a Brier-score calibration term to an RL reward and reports improved calibration without accuracy loss. Group Relative Policy Optimization (GRPO, arXiv:2402.03300) is the RL algorithm skeleton (group-relative baseline, no critic network) that both Laya’s “RLCD” and our proposed RL stage (§4, stage 3) build on. The companion calibration proposal in this graph develops this connection in full technical detail, specifically for small encoders.

Distillation. Hinton et al.’s soft-target distillation (arXiv:1503.02531) is the mechanism Maestro Flash’s stage-2 training uses to transfer distributional judgment from large decoders (Qwen3 arXiv:2505.09388, Gemma 3 arXiv:2503.19786) into the small encoder. The broader taxonomy in arXiv:2006.05525 informs what signal (final distribution or intermediate representations) to distill.

Datasets. MASSIVE (arXiv:2204.08582), cross-lingual natural language inference (XNLI, arXiv:1809.05053), and Banking77 (arXiv:2003.04807) are the three public benchmarks Laya reports against Jev. We adopt all three for direct comparability (§5) and add a fourth, new benchmark for Indic-language coverage (the Indic-first benchmark proposal). IndicXTREME (arXiv:2212.05409) is the closest existing Indic-specific multi-task benchmark, but it targets classification, natural language inference and question answering rather than the typed-decision primitive taxonomy this project uses.

3. Method: proposed architecture

3.1 Backbone

Maestro Flash’s primary backbone candidate is mmBERT-base (322M parameters as counted in Laya’s README, arXiv:2509.06888). It is chosen for its native multilingual coverage and because it is Laya’s own multilingual backbone, which gives a controlled reference point. A second candidate, Ettin-encoder-400M (arXiv:2507.11412) with continued pretraining on multilingual data, is proposed as an ablation (§4, stage 0) rather than assumed inferior. Ettin’s paired, controlled encoder and decoder training methodology is the strongest available evidence for encoder superiority on classification-shaped tasks. It is an open empirical question whether that recipe advantage survives continued multilingual pretraining better or worse than mmBERT’s native multilingual recipe.

3.2 choice head

Standard practice (also Laya’s) encodes state, question, and all k options jointly within a shared token budget, projecting each option’s pooled representation to a logit and taking a softmax over the k options. Laya’s own README attributes its clearest documented weakness to exactly this design. Its Banking77 accuracy is 0.425–0.492, against Jev’s published 0.870 (a third-party figure measured on a 72-label condition, so the two denominators differ). The cause: a fixed head_max_len token budget shared across all options starves high-cardinality (77-option) questions of per-option tokens. Laya’s README computes this explicitly: (256 - 16) // 77 ≈ 3–4 tokens per label. Maestro Flash’s choice head instead encodes option text through a separately batched, cacheable forward pass decoupled from the state’s context budget, so option-set cardinality no longer trades off against per-option token allocation. This is a proposed fix to a diagnosed, documented failure mode; §5 specifies Banking77 as the direct validation benchmark.

3.3 score head

An ordinal-regression (cumulative-link, or proportional-odds) output over rubric levels, rather than unordered softmax classification. Laya’s README identifies ordinal score questions as its weakest primitive (SST-5, the five-class Stanford Sentiment Treebank: 0.372), and its internal handling treats rubric levels as unordered classes. A cumulative-link head respects the ordering directly and yields an expected-value score from the fitted cumulative distribution rather than an ad hoc weighted average.

3.4 span and json heads

span: a start and end pointer network over the state’s token sequence (standard extractive question-answering architecture). It gives Maestro Flash an extraction primitive that neither Laya nor, as far as this review found, Jev exposes natively.

json: a target schema is decomposed into typed leaf fields. Each field is dispatched to the appropriate existing head (choice for enums, score for bounded numerics, noul for booleans, span for free text), predicted in parallel in one forward pass, and assembled deterministically into the output object. This is an application of the non-autoregressive structured-output framing surveyed in arXiv:2204.09269: the schema replaces the decoding grammar, and parallel typed decisions replace sequential token emission. Nested and recursive schemas (arrays of objects) are out of scope for v1 and are stated as a limitation (§7).

3.5 Multilingual and Indic pipeline

Training data for non-English languages is built in three steps. (1) Machine-translate English seed typed-decision examples into 11 major scheduled Indic languages using IndicTrans2 (arXiv:2305.16307). (2) Generate Latin-script transliterated variants, following MuRIL’s transliteration-augmentation method (arXiv:2103.10730), to cover common code-mixed real-world input. (3) Hold out a small, human-verified (not machine-translated) subset for evaluation. This held-out subset is the seed for the Indic-first benchmark proposal. Machine-translated training data risks training the model on translation-shaped rather than natively fluent text, the same risk IndicXTREME’s construction methodology (arXiv:2212.05409) was designed to expose for classification tasks generally.

4. Training recipe

Stage 0 (backbone selection). Ablate mmBERT-base against continued-pretrained Ettin-encoder-400M on a small typed-decision dev set, and select the stronger backbone before committing further compute.

Stage 1 (supervised multi-task fine-tuning). Train all four heads jointly on (a) public datasets reframed as typed-decision questions (MASSIVE, XNLI, Banking77, AG News and DAIR Emotion, the same set Laya’s own training mix draws from, per its README and BENCHMARKS description), (b) the Indic-augmented data from §3.5, and (c) a synthetic schema-conditioned JSON corpus generated by prompting a large teacher decoder (Qwen3 or Gemma 3). The synthetic corpus is needed because no public large-scale typed-JSON extraction corpus with verified ground truth was located during this review.

Stage 2 (distillation). Also train against the full output distribution of a large teacher decoder for each example (not just the hard label), following Hinton et al.’s soft-target method (arXiv:1503.02531). This directly targets Laya’s own documented weakness in soft accuracy (its README reports 0.471 against Jev’s published 0.580 on typed-decisions) by making distribution matching an explicit stage-2 objective.

Stage 3 (RL against proper scoring rules). GRPO-style RL (arXiv:2402.03300: group-relative baseline, no critic network) with a reward built from a strictly proper scoring rule (Brier score for choice and noul, following the loss-substitution method validated at decoder scale in arXiv:2405.18906), plus an explicit calibration term following RLCR’s approach of penalizing the gap between stated confidence and realized correctness (arXiv:2507.16806). The companion calibration proposal develops the full technical specification of this stage, including reward derivations and the connection to GRPO’s group-relative baseline.

Stage 4 (calibration). Fit one temperature per (question type, option count, language) on held-out data, following Guo et al. (arXiv:1706.04599) and replicating Laya’s own documented practice. The language dimension is added because Laya’s own per-language ECE table shows substantial cross-language variation at fixed question type and option count (for example, English ECE 0.179 against Khmer ECE 0.952 on the same MASSIVE-intent task, per checkpoint, in Laya’s BENCHMARKS.md).

5. Experimental plan

Metrics: accuracy, soft accuracy (distribution match to teacher), Brier score, ECE, ordinal mean absolute error (MAE), median (p50) and batched latency, and option-order flip rate. Laya’s own README flags flip rate as a weakness: its flip rates at 20 options trail the 0.13 it cites for Jev.

Benchmarks: MASSIVE (arXiv:2204.08582, all 51 languages, with an explicit Indic-language breakout), XNLI (arXiv:1809.05053), Banking77 (arXiv:2003.04807, the direct test of the choice head fix in §3.2), AG News, DAIR Emotion, and two more. The first is a typed-decisions application-workflow benchmark built with the same four-theme methodology Laya’s own benchmark uses (invoice processing, security incidents, customer service, agent-trace observability). The second is the new human-verified Indic typed-decision benchmark from the Indic-first benchmark proposal.

Go/no-go framing, stated in advance rather than post hoc:

  • Maestro Flash’s choice head should match or exceed Laya’s own reported Banking77 number (0.425–0.492), since it is a direct architectural fix to Laya’s own diagnosed cause of that number.
  • Maestro Flash’s calibration (ECE) should be evaluated per language, not only as a macro average, given Laya’s own documented per-language ECE spread.
  • Maestro Flash’s Indic-language accuracy should be reported primarily on the new human-verified benchmark, not solely on the existing Indic subsets of MASSIVE and XNLI, since those were not designed to test the choice/score/span/json primitive taxonomy.

6. Compute budget

8×H100 week (1,344 H100-hours total): backbone ablation (~96h), stage-1 supervised fine-tuning (~384h), teacher-distillation data generation (~192h, inference-bound), stage-2 distillation fine-tuning (~240h), stage-3 RL (~288h), stage-4 calibration and full evaluation suite (~96h), buffer (~48h). This is a planning allocation (the Maestro Flash architecture and roadmap gives the same table), not a measured cost, and it should be re-budgeted once stage-0 and stage-1 wall-clock numbers are available.

Single RTX-class GPU: at 322-400M parameters, the backbone is under 1GB in 16-bit floating point (fp16). That makes both batched inference and low-rank adaptation (LoRA) fine-tuning comfortably feasible on a single 24GB-class consumer GPU (RTX 4090/5090). Full-parameter fine-tuning is also feasible at modest batch size. Inference throughput is a target, not yet measured. Its floor is set by Laya’s own T4 numbers (103-332 questions/sec batched, per Laya’s README), since a modern consumer GPU has substantially more compute and memory bandwidth than a T4.

7. Limitations

This is a design proposal; no trained model exists. Specific risks:

  • The json head’s schema decomposition is untested on nested or recursive schemas and is scoped to flat schemas for v1.
  • The synthetic JSON training corpus has no large-scale human-verified ground truth.
  • Per-(question type, option count, language) calibration may not have enough held-out data per cell for the lowest-resource Indic languages, which would require a language-family fallback grouping.
  • The mmBERT-versus-Ettin backbone decision is genuinely open, and this paper does not prejudge it.
  • Every claim about Maestro Flash’s eventual performance in this paper is a target or a cited baseline, never a measurement. That distinction must be preserved in all follow-up reporting.

8. Conclusion

Maestro Flash is proposed as an open, Apache-2.0, encoder-based typed-decision model. It extends Laya’s choice/score/noul primitive set with span and schema-typed json outputs, is built on a single multilingual backbone with fine-grained calibration, and is evaluated with an explicit, human-verified, Indic-first benchmark. Every design choice in this paper traces either to a verified published result or to a specific, documented failure mode in Laya’s own public benchmarks. The next concrete step, specified in the Maestro Flash roadmap, is a 90-day build plan that starts with a minimal, reproducible training skeleton that runs on a central processing unit (CPU).

References

  1. Warner et al. “Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference.” arXiv:2412.13663
  2. Marone, Weller, Fleshman, Yang, Lawrie, Van Durme. “mmBERT: A Modern Multilingual Encoder with Annealed Language Learning.” arXiv:2509.06888
  3. Weller, Ricci, Marone, Chaffin, Lawrie, Van Durme. “Seq vs Seq: An Open Suite of Paired Encoders and Decoders.” arXiv:2507.11412
  4. He, Gao, Chen. “DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing.” arXiv:2111.09543
  5. Khanuja et al. “MuRIL: Multilingual Representations for Indian Languages.” arXiv:2103.10730
  6. Qwen Team. “Qwen3 Technical Report.” arXiv:2505.09388
  7. Gemma Team, Google DeepMind. “Gemma 3 Technical Report.” arXiv:2503.19786
  8. Gu, Bradbury, Xiong, Li, Socher. “Non-Autoregressive Neural Machine Translation.” arXiv:1711.02281
  9. “A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond.” arXiv:2204.09269
  10. “A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models.” arXiv:2508.08712
  11. Lou, Meng, Ermon. “Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution.” arXiv:2310.16834
  12. Nie et al. “Large Language Diffusion Models.” arXiv:2502.09992
  13. “Dream 7B: Diffusion Large Language Models.” arXiv:2508.15487
  14. Guo, Pleiss, Sun, Weinberger. “On Calibration of Modern Neural Networks.” arXiv:1706.04599
  15. Shao, Meng, Liu, Zhou. “Language Generation with Strictly Proper Scoring Rules.” arXiv:2405.18906
  16. “Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty.” arXiv:2507.16806
  17. “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.” arXiv:2402.03300
  18. Hinton, Vinyals, Dean. “Distilling the Knowledge in a Neural Network.” arXiv:1503.02531
  19. “Knowledge Distillation: A Survey.” arXiv:2006.05525
  20. FitzGerald et al. “MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages.” arXiv:2204.08582
  21. Conneau et al. “XNLI: Evaluating Cross-lingual Sentence Representations.” arXiv:1809.05053
  22. Casanueva, Temčinas, Gerz, Henderson, Vulić. “Efficient Intent Detection with Dual Sentence Encoders.” arXiv:2003.04807
  23. “Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages.” arXiv:2212.05409
  24. “IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages.” arXiv:2305.16307
  25. Laya (github.com/NandhaKishorM/laya), README.md and BENCHMARKS.md, read directly, 2026. Apache 2.0. Third-party Jev benchmarks cited therein: github.com/AbdelStark/jev-benchmarks, github.com/nibzard/decision-model-benchmark.

← All papers