Calibrated Decisions via Proper-Scoring-Rule RL

Status: TECHNICAL PROPOSAL WITH PLANNED EXPERIMENTS. No experiment described in this paper has been run. Every equation is a proposed training objective; every number is either a cited published result, explicitly attributed, or a TARGET marked as such. This paper does not report any Maestro Flash result, because Maestro Flash has not been trained.

velofy.co research report, r3. 2026-09-23.

Abstract

Small encoder models trained for typed decisions (choose one of k labels; produce a bounded score; output a calibrated boolean probability) are usually trained with cross-entropy and then patched for calibration after training with temperature scaling. This two-stage approach is exactly what the open project laya (github.com/NandhaKishorM/laya) documents doing, and it works. Laya reports moving mean expected calibration error (ECE) from 0.466 to 0.081 on its English checkpoint through post-hoc temperature fitting. But it treats calibration as a correction applied after training, not as an objective trained for directly.

We propose training the typed-decision heads of small encoders with reinforcement learning (RL) against strictly proper scoring rules: loss functions that are uniquely minimized, in expectation, by reporting one’s true belief. The optimizer is a group-relative policy gradient in the style of Group Relative Policy Optimization (GRPO). The proposal follows two precedents. Reinforcement Learning with Calibration Rewards (RLCR, arXiv:2507.16806) adds a calibration reward at decoder scale, and Shao et al. (arXiv:2405.18906) substitute proper scoring rules for the training loss. We adapt both to the small, non-autoregressive, closed-output-space setting that encoder-based typed-decision models occupy.

We give the derivation connecting the Brier score, the spherical score, and GRPO’s group-relative baseline. We specify a reward decomposition that separates decision accuracy from calibration quality, following RLCR’s approach at decoder scale but reformulated for a closed-form softmax or ordinal output rather than a free-text confidence statement. And we lay out an experimental plan, with laya’s own published numbers as the baseline to match or exceed, for testing whether this approach reduces the need for post-hoc temperature correction, not just the residual ECE after correction.

1. Introduction

Calibration is the property that a model’s stated confidence matches its empirical accuracy. For typed-decision systems it is not a nice-to-have metric. It is often the mechanism by which a production pipeline decides whether to trust an automated decision or escalate it to a human. Laya’s own README makes this role explicit with a code example:

if conf >= 0.85:
    route_automatically(dept)
else:
    escalate_to_human_agent(dept, reason=f"Low confidence ({conf:.2f})")

If conf is not honest, this pattern fails silently in one of two ways: over-automation (acting confidently on wrong decisions) or under-automation (escalating decisions the model actually got right). Both are costly, and neither is visible from accuracy alone. A model can have excellent accuracy and terrible calibration at the same time. Laya’s own BENCHMARKS.md documents a sharper version of the same failure for its English checkpoint on non-Latin scripts: 0.000 accuracy at 0.952 mean confidence on Khmer. Confidence gating cannot catch this failure, because the confidence itself is what is broken.

The standard remedy is cross-entropy training followed by post-hoc temperature scaling (arXiv:1706.04599). It is cheap, and laya’s own numbers show it works well as a correction: one fitted temperature per (question type, option count) moves mean ECE from 0.466 to 0.081 on laya’s English checkpoint. That leaves a question open. If the model’s raw confidence is off by an amount a single scalar temperature can mostly fix, why not train the model so the correction is unnecessary in the first place? Or train it to a state where the residual miscalibration after correction is below 0.081, rather than accepting 0.081 as the floor?

This paper proposes an answer: train the typed-decision heads directly against a strictly proper scoring rule, through RL, rather than against cross-entropy. The idea is not new in isolation. Shao et al. (arXiv:2405.18906) show that substituting the loss (cross-entropy replaced by the Brier or spherical score) improves decoder-scale language generation. RLCR (arXiv:2507.16806) shows that adding a Brier-score reward term to RL improves calibration for free-text confidence statements from reasoning models. What is new here is the setting: small, non-autoregressive encoders with closed, typed output spaces. There, the “confidence statement” is not a token the model chooses to emit, as in RLCR’s free-text setting. It is a first-class softmax or ordinal output that is always present and always readable as a probability. This structural difference changes both the reward design (§4) and what “proper scoring rule” means operationally (§3).

Calibration. Guo et al. (arXiv:1706.04599) established that modern deep networks are systematically overconfident and that temperature scaling is a strong, cheap post-hoc fix. Laya uses this method, per its own documentation. Our proposal does not discard temperature scaling: §5 keeps it as a final stage, and asks whether RL against a proper scoring rule can shrink the correction it has to make.

Proper scoring rules in language model training. Shao et al. (arXiv:2405.18906) demonstrate, at decoder scale up to 13 billion parameters, that training language generation against non-local strictly proper scoring rules (Brier, spherical) instead of the logarithmic score (standard cross-entropy) yields “substantial improvements” with no other hyperparameter changes. This is direct evidence, at a much larger scale than this paper targets, that loss substitution works for language models in general. It motivates carrying the idea down to small encoders and typed decisions.

Calibration via RL reward shaping. RLCR (arXiv:2507.16806) is the closest precedent to this proposal, and the one laya’s “RLCD” terminology most plausibly echoes. We found no technical paper on arXiv documenting laya’s specific RLCD implementation; laya’s public documentation is the only primary source for its method. RLCR trains a reasoning decoder to emit both an answer and a numeric confidence after reasoning, and adds a Brier-score term to the RL reward that penalizes the gap between stated confidence and realized correctness. The paper reports that this improves calibration substantially without degrading accuracy, and that choosing among several sampled generations by self-reported confidence improves both accuracy and calibration as compute scales. RLCR’s setting is free-text confidence elicitation from an autoregressive reasoning model. Ours is closed-form probability output from a non-autoregressive encoder. That removes the need to parse a confidence statement out of generated text, but it raises a new question: how to define the reward over a whole probability distribution (a softmax over k options) rather than over one scalar confidence (§4).

RL algorithm. GRPO (arXiv:2402.03300) removes the separate critic (value) network used in Proximal Policy Optimization (PPO) style RL. It estimates a baseline from the relative scores of a sampled group of outputs, which substantially reduces RL compute and memory overhead. This is the algorithm skeleton we adopt (§4.3). In our proposal, the reward it optimizes is the strictly proper scoring rule described in §3, not a binary correctness signal.

The system this paper responds to. Laya (github.com/NandhaKishorM/laya) is the concrete, documented system whose published numbers this paper treats as the baseline to explain and, where possible, to improve on in a future trained model. Every laya number cited here comes directly from laya’s own README.md and BENCHMARKS.md.

3. Strictly proper scoring rules: background

A scoring rule $S(p, y)$ assigns a real-valued score to a probabilistic forecast $p$ given the realized outcome $y$. A scoring rule is proper if, for a forecaster whose true belief is a distribution $q$, the expected score $\mathbb{E}_{y \sim q}[S(p, y)]$ is optimized (say, minimized, under a loss-style sign convention) by reporting $p = q$. It is strictly proper if $p = q$ is the unique optimizer. This is exactly the property a calibration-honest training objective needs. If the loss rewards only honest probability reports, a model trained to convergence against it has no incentive to overstate or understate its confidence, independent of any post-hoc correction.

The standard cross-entropy loss used to train classifiers is, in expectation, the logarithmic score, which is strictly proper. This well-known fact is sometimes underappreciated: the usual classification loss is already a proper scoring rule. The practical problem, as Guo et al. (arXiv:1706.04599) document, is that the logarithmic score’s gradient landscape, combined with modern network capacity and optimization dynamics, still tends to produce overconfident models in practice. Properness in expectation does not guarantee good calibration at the finite-sample, early-stopped, regularized optimum that real training reaches. Other strictly proper scoring rules behave differently on finite samples. The two most relevant alternatives for this proposal:

Brier score. For a k-way categorical forecast $p = (p_1, …, p_k)$ and one-hot outcome $y$: $S_{\text{Brier}}(p, y) = \sum_{i=1}^{k} (p_i - y_i)^2$. This is the reward term RLCR uses (arXiv:2507.16806) and the metric laya reports throughout its benchmarks (for example, laya’s typed-decisions Brier of 0.062 against Jev’s published 0.148). Because it is already laya’s main calibration-adjacent metric, using it as the reward and not only as the evaluation metric is a direct design choice: optimize the thing you will be measured on.

Spherical score. $S_{\text{sph}}(p, y) = \frac{p_y}{|p|_2}$ for the realized class $y$, normalized differently from the Brier score. Shao et al. (arXiv:2405.18906) report the spherical score as a viable alternative to the Brier score for decoder-scale language generation.

One theoretical point matters for reward design under group-relative RL. We found it in search results on the geometry of proper scoring rules under group-relative optimization during this review, and cite no paper for it here. Different proper scoring rules are equivalent in expectation, but once embedded in a group-relative baseline like GRPO’s they can give meaningfully different finite-sample learning signals. So the choice between the Brier and spherical score is not cosmetic when RL, rather than direct gradient descent on the full-batch expectation, does the optimizing. This is why §5 treats the scoring rule as a hyperparameter to sweep, not a settled choice.

4. Method

4.1 Setting

We consider a typed-decision head $f_\theta$ (encoder trunk plus task head, following the architecture in the Maestro Flash proposal). Given input state $x$ and a typed question $q$ (with an explicit option set for choice, a rubric for score, or nothing extra for noul), it outputs a probability distribution $p_\theta(\cdot \mid x, q)$ over the closed output space that $q$ defines. This is a closed-form quantity, unlike the free-text confidence elicitation RLCR addresses. There is no parsing step, and the distribution always sums to one by construction: a softmax for choice and noul, and a distribution derived from a cumulative link for ordinal score, per the head design in the Maestro Flash technical report.

4.2 Reward decomposition

We propose a two-term reward. It follows RLCR’s separation of accuracy and calibration (arXiv:2507.16806), reformulated for a closed-form distribution rather than a single scalar confidence:

$$ R(x, q, y) = \underbrace{-S(p_\theta(\cdot \mid x, q), y)}{\text{proper scoring term}} + \lambda \cdot \underbrace{\mathbb{1}[\arg\max p\theta = y]}_{\text{decision-accuracy bonus}} $$

Here $S$ is the Brier score (§3), chosen as primary because it corresponds directly to laya’s existing reported metric, and $\lambda$ is a tunable weight on an explicit argmax-correctness bonus. The proper scoring term alone already rewards correctness and calibration jointly, since a proper scoring rule is minimized by the honest, correct distribution. But RLCR’s own ablations at decoder scale motivate keeping an explicit accuracy term as well. Pure calibration objectives can, in principle, be satisfied by a well-calibrated but low-confidence, low-accuracy model. A model that always outputs the training-set base rate, for example, is reasonably well calibrated in aggregate but useless as a decision system. This joint objective is a design choice to be validated empirically (§5), not a theorem.

4.3 GRPO-style optimization

Following GRPO (arXiv:2402.03300): for each training example $(x, q, y)$, sample a group of $G$ stochastic forward passes and compute the reward $R_g$ for each. Because the head is not autoregressive and has no sampling temperature in the usual decoder sense, the variation comes from input-side augmentation (option-order permutation, per §4.4) or from dropout in the trunk. The group-relative advantage is:

$$ A_g = \frac{R_g - \text{mean}(R_1, …, R_G)}{\text{std}(R_1, …, R_G) + \epsilon} $$

The policy gradient update follows GRPO’s clipped surrogate objective, using $A_g$ in place of a learned critic’s advantage estimate. This removes the need for a separate value network, matching GRPO’s original motivation of cutting RL memory and compute. That matters here because the target model is a small encoder (300 to 500 million parameters), where a separate critic of comparable size would roughly double training memory for no benefit specific to this setting.

One structural difference from GRPO’s original setting, verifiable math reasoning, needs stating. There, each sample in the group is a full autoregressive rollout: different reasoning paths to different answers. In our non-autoregressive setting, the natural source of variation within a group is not sampled reasoning paths but input-side augmentation of the same example: permuted option order, back-translated or transliterated phrasings of the same state (for multilingual robustness, tying into §4.4), or dropout in the forward pass. This is a genuine adaptation of GRPO to a setting it was not designed for. Whether group-relative advantage estimation still gives a useful, low-variance training signal under this different source of variation is an open empirical question, and this paper flags it rather than assuming it.

4.4 Option-order robustness as a training signal

Laya’s own README documents real sensitivity to option order: flip rates of 0.15 to 0.23 at 20 options on its own benchmarks, against Jev’s published 0.13. It implicitly attributes this to too little option-order randomization during training. We propose making option-order permutation an explicit part of the group sampling in §4.3. Each group member for a choice question is the same example with a different random option order. The reward adds an explicit consistency penalty: a divergence term (for example, mean pairwise Kullback-Leibler (KL) divergence) between the group members’ predicted distributions, after mapping each back to a canonical option order. It enters the reward decomposition in §4.2 with its own weight $\mu$. This targets the option-order problem identified in laya’s documentation and in the typed-decisions survey directly, by making order invariance an explicit training target rather than something hoped to emerge from standard training.

5. Experimental plan

Baselines, stated in advance. All experiments compare against (a) standard cross-entropy training followed by post-hoc temperature scaling (the approach laya documents, replicated as a same-backbone baseline) and (b) laya’s own published numbers directly, where the same public benchmark (typed-decisions, MASSIVE, cross-lingual natural language inference (XNLI), Banking77) is used.

Metrics. ECE before and after any additional temperature refit; Brier score; accuracy; and, specific to §4.4, option-order flip rate. The key question is whether RL training reduces the size of the post-hoc correction needed. That is measurable as the gap between pre-refit and post-refit ECE shrinking relative to the cross-entropy baseline.

Ablations. (1) Brier against spherical score as the proper scoring term in §4.2; (2) with and without the explicit accuracy bonus; (3) with and without the option-order consistency penalty (§4.4); (4) a sweep over group size $G$, since GRPO’s group-relative variance reduction is sensitive to group size and this paper’s non-autoregressive group construction (§4.3) is untested at any group size.

Go/no-go criterion, stated in advance. The method counts as validated only if it achieves lower post-refit ECE than the cross-entropy-plus-temperature-scaling baseline at matched accuracy. Improving calibration by giving up accuracy, or the reverse, would not support the core claim: that training directly against a proper scoring rule is a better use of the same compute than training against cross-entropy and correcting afterward. No target ECE value is stated here. The right target is relative to the cross-entropy baseline actually measured on the same data and backbone, not an absolute number asserted in advance.

6. Limitations

This proposal has not been validated. Specific risks:

The group construction in §4.3 (input augmentation rather than sampled reasoning paths) departs from GRPO’s original design, and its variance-reduction properties are unverified in this setting.
The reward decomposition in §4.2 adds two hyperparameters ($\lambda$, $\mu$) whose sensitivity is unknown.
RL training is known, in the broader literature, to be less sample-efficient and more hyperparameter-sensitive than supervised fine-tuning. That may make this approach more expensive in practice than the compute budget in the Maestro Flash technical report (§6) assumes. That budget gives this stage about 288 of its 1,344 H100-hours (an 8×H100 week), the second-largest share after supervised fine-tuning at about 384.

7. Conclusion

We propose training small, non-autoregressive typed-decision encoders directly against strictly proper scoring rules with a GRPO-style group-relative RL objective. The approach adapts decoder-scale precedents (Shao et al.; RLCR) to a closed-form, non-autoregressive output setting, and adds an explicit reward term aimed at the option-order weakness laya’s own documentation identifies. It is fully specified but entirely untested. The experimental plan in §5 is the next step, to be run as part of the Maestro Flash stage-3 training run described in the Maestro Flash technical report.

References

Guo, Pleiss, Sun, Weinberger. “On Calibration of Modern Neural Networks.” arXiv:1706.04599
Shao, Meng, Liu, Zhou. “Language Generation with Strictly Proper Scoring Rules.” arXiv:2405.18906
“Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty” (RLCR). arXiv:2507.16806
“DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models” (GRPO). arXiv:2402.03300
Marone, Weller, Fleshman, Yang, Lawrie, Van Durme. “mmBERT: A Modern Multilingual Encoder with Annealed Language Learning.” arXiv:2509.06888
Weller, Ricci, Marone, Chaffin, Lawrie, Van Durme. “Seq vs Seq: An Open Suite of Paired Encoders and Decoders.” arXiv:2507.11412
Laya (github.com/NandhaKishorM/laya), README.md and BENCHMARKS.md, read directly, 2026.
Brier, G.W. (1950). “Verification of Forecasts Expressed in Terms of Probability.” Monthly Weather Review, 78(1). (Non-arXiv; foundational, pre-dates arXiv.)
Gneiting, T. & Raftery, A.E. (2007). “Strictly Proper Scoring Rules, Prediction, and Estimation.” Journal of the American Statistical Association, 102(477). (Non-arXiv.)

← All papers