Laya: An Open Typed-Decision Engine, Examined

Laya is an open, Apache-2.0 typed-decision engine by Nandakishor M. It uses small, bidirectional encoder models to answer predeclared questions about text or JavaScript Object Notation (JSON) state without autoregressively generating an answer. This teardown reads its public code and documentation closely, separates the project’s self-reported figures from independently run observations, and credits the engineering where it is warranted.

First principles

A typed decision has three parts: a state to read, a set of outputs declared in advance, and a probability over that set. Most of what follows in this teardown, the strengths and the failure modes, comes from four constraints on that shape.

• A declared output set turns generation into scoring. The model ranks options it was given instead of writing an answer token by token. The next section shows what that buys.
• The encoder reads a fixed number of tokens. Instructions, options and state compete for the same window. Every option added takes room from the others and from the state, so high option counts are a structural limit, not a tuning problem.
• A softmax always sums to one. The model must spread all its probability over the options it was shown, even when none fits or the input is in a script it cannot read. High confidence can mean only that one option looked less wrong than the rest.
• Calibration and ranking are separate properties. Dividing every logit by the same positive temperature keeps their order, so the top answer never changes. A temperature can make confidence honest enough to gate on; it cannot make a wrong answer right.

A fifth point governs the method: a project’s own figures are claims until someone reruns them. This teardown reads the code, reports the project’s numbers as self-reported, and adds one independent local run.

What Laya does

Laya accepts a state and a set of typed questions in one of three forms:

The output space is declared before inference. The model therefore selects from the declared options rather than emitting JSON or labels as tokens. That makes malformed output structurally impossible for the closed primitives, and lets all questions in a call be placed into one batch and passed through the encoder once.

The English checkpoint uses ModernBERT-large, 421M parameters in Laya’s accounting; the multilingual checkpoint uses mmBERT-base, 322M. Both are encoder-only transformers. Laya truncates inputs to 512/1024 tokens despite the backbones’ 8k native context. It also provides a Router, which selects a checkpoint from script and language heuristics.

Architecture

For a choice question, Laya builds a sequence shaped like [CLS] <qtype> instructions [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] state [SEP]. Each option gets a [MASK] marker. The encoder produces a representation for every token, a small 2-layer TransformerEncoder refines it, then a scorer reads each marker vector and returns one logit per option. A softmax over those logits is the probability distribution.

This is a credible, compact way to make an encoder answer arbitrary option sets in a single forward pass. It also explains a documented failure mode: options and instructions share a fixed head_max_len budget. With 77 labels, Banking77 labels receive roughly 3-4 tokens each by default, so they become difficult to distinguish. The repository’s predict_shortlist offers embedding-based pre-filtering, while manually increasing head_max_len and max_len trades away state context.

The model adds a type embedding and has an act_head, a small Linear(d+4,256)→GELU→Linear(256,n_act) classifier. It consumes the pooled [CLS] representation plus top-1 probability, top-1/top-2 gap, normalized entropy, and k/255, returning answers[qid]["action"]["act_probability"]. This is a distinct output from Laya’s documented entropy confidence and is not explained in the README or benchmarks.

The project also does several pragmatic things well. Narrow snapshot_download patterns avoid unnecessary checkpoint downloads. Compatibility is checked against parameter prefixes and shapes before a strict load. Fallback across NVIDIA’s Compute Unified Device Architecture (CUDA), Apple’s Metal Performance Shaders (MPS) and the central processing unit (CPU) is explicit. reference_compile = False avoids ModernBERT’s small-batch compile overhead. The router uses dependency-free Unicode and Latin-language heuristics, a least-recently-used (LRU) checkpoint cache, and an RLock that protects model lifecycle without serializing inference. The code explicitly avoids defaulting an unidentified Latin language to English.

RLCD and calibration

Laya calls its training method RLCD, meaning reinforcement learning against strictly proper scoring rules. Its public proper_reward combines log score, 0.5 times spherical score, and for ordinal questions subtracts 1.0 times ranked probability score. This is not the academic RLCD of Yang et al., arXiv:2307.12950, which means Reinforcement Learning from Contrastive Distillation. The actual REINFORCE or Group Relative Policy Optimization (GRPO) style training loop is in a Kaggle notebook rather than the installed package, which is inference-only.

At inference, Laya fits one temperature per (question_type, option_count_bucket) bucket: 2, 3-5, 6-10, and 11+. Temperatures are clamped to [0.5, 5.0]. This is a thoughtful safeguard. A shipped choice:11+ value of 0.1006 would sharpen logits by about 10×; the client clamps it instead of silently publishing the sharpened confidence. Entropy confidence is 1 - H(p)/log(k), not maximum probability.

What the public benchmarks say

The following figures are Laya’s own published measurements, not results independently reproduced here.

• Base checkpoints are near chance on its typed-decisions benchmark: 0.362 / 0.342, against a 0.318 random floor and a 0.461 majority-class floor. The headline 0.766 is from fine-tuning on that benchmark’s training split.
• Banking77 is 0.425 for both English and multilingual checkpoints. The project attributes this to the option-token budget described above.
• The English checkpoint has Khmer accuracy 0.000 at mean confidence 0.952; confidence never falls below 0.885 at any accuracy level. Routing must happen before the model call.
• Mean as-shipped expected calibration error (ECE) is 0.466 for laya and 0.314 for laya-multilingual; the multilingual model has no fitted temperatures. score is the weakest primitive, with the five-class Stanford Sentiment Treebank (SST-5) at 0.372.
• On two reported Jev comparisons, Laya reports soft accuracy 0.471 versus 0.580, and raw ECE 0.213 versus 0.144. Held-out toxic-chat is 0.530 accuracy and 0.400 macro-F1.
• At 20 options, option-order flip rates are 0.150/0.230, compared with Jev’s reported 0.13.

Those disclosures are useful because they make the limits inspectable. There are also clear reasons not to overgeneralize from the headline number. It comes from in-distribution fine-tuning, the high-cardinality limitation is architectural, and all Jev comparison figures in Laya’s README and BENCHMARKS.md are self-reported from third-party publications rather than a live comparison against TypeSafe’s application programming interface (API).

The repository has 1,729 test lines across 8 files. Offline tests cover tensor-shape and single-option cases, router precedence, language detection, shortlisting, packaging, download behavior, and local end-to-end loading. That is a real safety net for a young package, though it does not substitute for a reproducible training implementation.

Apple Silicon smoke test

We installed laya 0.3.6, torch 2.14.0, and transformers 5.17.0 in a fresh Python 3.12.12 environment on macOS Darwin 25.3.0, Apple Silicon. The English checkpoint was 842,609,210 bytes, about 803 MB; the multilingual checkpoint was 643,835,514 bytes, about 614 MB.

The agent selected MPS. Cold construction took 28.66 s. Across 10 timed calls after one warmup, each containing three questions, choice plus score plus noul, mean latency was 79.6 ms, minimum 78.7 ms, maximum 82.1 ms. This is a real run on a different accelerator from the project’s NVIDIA T4 figures, so it is not a like-for-like benchmark. It does establish that the software development kit (SDK) performed a batched, nontrivial forward pass on an Apple Silicon Mac.

The live run also reproduced two concrete conditions. The load warning reported that choice:11+=0.1006 was outside [0.5, 5] and had been clamped. More importantly, act_probability was 1.0 for all three answers, including an urgency score with confidence 0.1425 and distribution [0.116, 0.327, 0.557]. This matches issue #185, which reports unnormalized decision-head outputs saturating act_probability. Until it is resolved, action.act_probability should not be used as an automation gate. Laya’s documented entropy-confidence gating pattern is separate and unaffected by this finding.

Limits and provenance

The repository was created 2026-09-18T04:46:33Z; the surrounding research observed 17,668 stars, 1,503 forks, 138 open issues, about 121 pull requests, and 16 point releases on the Python Package Index (PyPI) in its first 5 days. Those are attention signals, not validation. The author’s priority claim links arXiv:2510.01237, whose abstract concerns pre-generation hallucination routing rather than RLCD, proper scoring rules, or non-autoregressive decisions. That does not diminish the project’s visible engineering or the reproducible smoke test; it simply leaves the priority argument weaker than the product and code review.

Laya is a serious open attempt at a useful interface: state in, typed probabilities out. Its code makes the one-pass claim concrete, its calibration clamp is a sound defensive choice, and its documentation names several of its own weaknesses. The parts most worth testing next are high-cardinality options, multilingual calibration, the undocumented action head, and whether the training recipe can be independently reproduced.

← All papers