Why Option Order Flips Typed Decisions, and How to Measure It

The question

A typed-decision model takes an input state and a declared set of options, then returns a choice with probabilities. The options form a set, so their presentation order should not matter. In practice it does: reorder the same options and the model sometimes returns a different answer. This report asks why that happens, how to measure it, and what fixes exist. It covers evidence from multiple-choice studies of large language models (LLMs) and from the small typed-decision systems tracked in this graph. It ends with a measurement design that a small-model project can adopt directly.

Two facts from this graph are taken as established and are not re-argued here. The Jev survey reports a third-party measurement in which Jev flips 13% of its choices under option permutation. The Laya teardown describes Laya as an open encoder model that scores one marker per option in a single forward pass, and Laya’s own benchmark document reports higher flip rates than Jev’s at 20 options. This report builds on those facts instead of repeating them.

Background

Order sensitivity has at least three distinct entry points. Keeping them separate matters, because each implies a different fix.

First, option identifier (ID) bias. When a decoder LLM answers a multiple-choice question, it usually emits a label token such as A, B, C, or D. If the model assigns unequal prior probability to those tokens, answers drift toward favored letters regardless of content. Zheng et al. (arXiv:2309.03882) name this token bias and identify it as the primary source of what they call selection bias. This mechanism is specific to systems that predict option IDs as tokens.

Second, position bias. Even without letter labels, a model can favor options by position, for example the first or last item in a list. Judge-style evaluations show this clearly: the same two responses in swapped positions get different verdicts. Zheng et al. (arXiv:2306.05685) list position bias alongside verbosity and self-enhancement biases in LLM judges, and Wang et al. (arXiv:2305.17926) show that reordering alone can reverse judge verdicts. Any model that reads options as an ordered sequence can carry this bias, including encoder heads.

Third, sequence-order leakage in joint encoding. Encoder decision heads typically encode the state and all options in one sequence, so absolute position embeddings and directional attention patterns differ under permutation. This graph’s Laya teardown describes exactly such a scheme, with one marker per option inside a shared sequence. Nothing in a standard Transformer forces the per-option scores to be invariant to option order, so some sensitivity is the default, not a surprise.

The clean formalism comes from the sets literature. Zaheer et al. (arXiv:1703.06114) characterize the family of permutation-invariant functions and give necessary and sufficient conditions for equivariance in deep models. Lee et al. (arXiv:1810.00825) build an attention module, the Set Transformer, designed so set-structured outputs do not depend on input order. A typed-decision head is, mathematically, a function from a set of options to a distribution. The gap between that specification and the sequence-based implementation is where flips live.

What the evidence says

The table below summarizes the primary sources fetched for this report. LLM multiple-choice studies come first, then judge studies, then mitigation papers, then typed-decision measurements.

Several patterns stand out. First, the effect is large wherever it is measured carefully. Pezeshkpour and Hruschka report gaps from approximately 13% to 75%, Wang et al. report verdict reversals on most tested queries, and Alzahrani et al. report that leaderboard positions move by up to 8 places. Second, typed-decision systems are not exempt. The independent benchmark by nibzard reports Jev at 13% flips, and Laya’s own document reports its checkpoints at 0.150 and 0.230 on the intent suite, explicitly worse than the 0.13 it cites for Jev. Third, the proposed causes converge: Pezeshkpour and Hruschka conjecture the sensitivity bites when the model is uncertain between its top 2 or 3 choices, while Zheng et al. (arXiv:2309.03882) locate the prior in option ID tokens. Uncertainty plus a positional or token prior is the common story.

How to measure order sensitivity

A usable protocol needs one primary metric, one distributional metric, and enough discipline about randomization that two labs get the same number.

The primary metric is flip rate: the fraction of items whose top-1 answer changes across permutations of the options. Both typed-decision sources in this graph already use it. The nibzard benchmark reports Jev changing 13% of choices on reruns with permuted order, and Laya’s document defines its table as how often the answer changes when options are permuted. Flip rate is easy to explain and directly answers the operational question, which is whether a user who shuffles options gets a different decision.

The distributional metric should capture probability movement even when the top answer survives. Two label-free options exist in the fetched literature. Guda et al. propose the Permutation Bias Metric, which quantifies inconsistency of predictions across answer orderings without needing labels. Choi et al. propose Choice Kullback-Leibler Divergence, which captures distributional imbalance in predictions. Either is preferable to accuracy-under-permutation alone, because accuracy can stay flat while confidence swings, and confidence is part of the typed-decision contract.

Four protocol rules make the numbers comparable. First, fix the permutation set in advance with recorded seeds, and report the seed list, so a rerun reproduces the exact permutations. Second, un-permute predicted distributions back to a canonical option ordering before comparing them, otherwise a correct equivariant model looks inconsistent. Third, stratify by option count, because Laya’s own data shows the problem growing with cardinality: its intent suite at 20 options is where both its checkpoints trail Jev. Fourth, report accuracy on the permuted suite alongside flip rate, following the nibzard design, which keeps a dedicated permuted suite (its S4 column) rather than folding permutation into the main accuracy number.

A minimal design that satisfies all four: for each item, evaluate the identity order plus a fixed number of seeded random permutations, record top-1 per permutation and the full distribution in canonical order, then report flip rate, mean pairwise divergence across permutations, and permuted-suite accuracy, each stratified by option-count bucket. No new infrastructure is needed beyond seeded shuffling and canonical reordering.

Analysis: why order changes answers

Three mechanisms, matching the three entry points in the background section, explain most observed flips. They are not mutually exclusive, and a given model can exhibit all three.

Token priors explain decoder LLM selection bias. Zheng et al. (arXiv:2309.03882) report that models assign more prior probability mass to specific option ID tokens such as A, B, C, or D, and that this prior survives into the final prediction. Their PriDe method works precisely because the prior can be estimated separately: permute option contents across a small number of test samples, measure the residual preference for each ID, and subtract it from later predictions. The fact that a content-independent prior can be estimated and removed is strong evidence that the prior is real and separable. This mechanism should not apply to encoder heads that never predict ID tokens, which makes those heads an interesting control group: whatever flips remain there must come from the other two mechanisms.

Uncertainty gates the damage. Pezeshkpour and Hruschka conjecture that sensitivity arises when the model is uncertain between its top 2 or 3 choices, with specific placements favoring one contender over another. This predicts the observed heterogeneity: easy items with a dominant option rarely flip, while close calls flip often. It also predicts that flip rate and calibration interact. A model that reports honest uncertainty on close calls will still flip its top-1 under permutation, but its distributions will move less than those of an overconfident model that swings from one certainty to another. This is why the distributional metric matters: two models with the same flip rate can differ greatly in how much their probabilities move.

Sequence position leaks into option scores. Standard encoders add position information to every token, and attention patterns can encode primacy or recency preferences learned from pretraining data. In a marker-per-option scheme like the one this graph’s Laya teardown describes, permuting options moves each option’s marker to a different absolute position with different neighbors, so the scores change unless training has taught the head to ignore those signals. Laya’s own benchmark document points at exactly this cause when it prescribes more aggressive option-order shuffling during training. Note the asymmetry with token priors: shuffling during training can plausibly erase position leakage, because the head sees every option in every position, but it cannot erase an ID-token prior in a decoder that always emits IDs in the same slots.

One more interaction deserves attention. Per-bucket temperature calibration, which Laya fits per question type and option-count bucket, rescales logits symmetrically and therefore cannot remove order dependence. It can only shrink or stretch the confidence of whatever the order-dependent logits already say. Calibration and order robustness are separate axes, and a project that fits temperatures on unpermuted data may even bake one arbitrary ordering into its shipped confidence mapping.

What fixes exist

Fixes fall into four families: inference-time corrections, training-time objectives, architectural invariance, and weight editing. Each has a different cost profile.

Inference-time corrections need no retraining. The simplest is permute and vote: run several option orders and take the majority answer. Guda et al. report that majority voting is effective but computationally prohibitive in its naive form, and propose Batch Question-Context key-value caching to cut that cost. PriDe, from Zheng et al. (arXiv:2309.03882), is cheaper: it estimates one ID prior from permuted contents on a small sample and applies it to the rest, with no labels required. Balanced Position Calibration, one of three strategies from Wang et al., aggregates results across orders to set the final score. Choi et al. add Auxiliary Option Injection, which introduces an extra answer choice to absorb bias in both white-box and black-box settings. All of these multiply inference cost or complexity, which matters most for systems whose selling point is low latency.

Training-time fixes aim to remove the bias at the source. The cheapest is permutation augmentation: shuffle option order during training so the head cannot rely on position. This is the fix Laya’s own document names for its 20-option weakness. Guda et al. propose an unsupervised LoRA fine-tuning strategy built on their permutation-consistency metric. The most elaborate training fix in the fetched set is PA-GRPO from Zheng et al. (arXiv:2603.21016): build a permutation group per instance, compute advantages relative to the mean reward over all permutations of that instance (cross-permutation advantage), and add a consistency-aware reward for agreement across permutations. They report outperforming strong baselines across 7 benchmarks while cutting selection bias. A sibling calibration proposal in this graph independently sketches the same shape for small encoders, with a pairwise divergence penalty across permuted group members, which suggests the idea is converging from both the LLM and small-encoder directions.

Architectural invariance removes the problem by construction. Deep Sets (arXiv:1703.06114) characterizes the invariant function family, typically implemented as per-element encoding followed by a symmetric pool. The Set Transformer (arXiv:1810.00825) adds attention over set elements with an inducing-point scheme that cuts self-attention cost from quadratic to linear in set size. For typed decisions, the practical analogue is decoupled option encoding: encode each option independently of the others and of absolute position, then score. This graph’s Maestro Flash proposal names exactly that design for its choice head, motivated by Laya’s high-cardinality failure. Whether decoupled encoding actually delivers invariance at 77-way scale is UNVERIFIED, since the proposal has no trained weights, but it is the most directly testable architectural hypothesis in the graph.

Weight editing is represented by Bias Node Pruning from Choi et al., which removes parameters identified as contributing to selection bias. It is the least explored family here and the hardest to audit, since the pruned parameters have no guaranteed interpretation. It is listed for completeness, not recommended.

Implications for small decision models

For Jev, the implication is measurement, not repair, because Jev is a closed API with no public weights or training paper. The nibzard benchmark reports 13% flips with all logs published, and notes that vendor evaluations do not measure this axis. A team that depends on Jev should therefore treat 13% as a planning input: replicate the permuted rerun on its own label sets, since flip rates vary by suite in every source that stratifies them, and consider client-side permute and vote for high-stakes calls. Client-side voting multiplies the per-decision cost and latency that motivate Jev adoption in the first place, so the decision is genuinely a tradeoff, and it cannot be made without the project’s own flip-rate number.

For Laya, the implication is a concrete training change. Its benchmark document is unusually honest: both checkpoints trail the cited Jev figure at 20 options, and the prescribed remedy is more aggressive option-order shuffling during training. Because Laya is open source with public checkpoints, this hypothesis is directly testable by anyone: retrain or fine-tune with shuffling, rerun the same three suites, and check whether the intent-suite numbers move toward the cited 0.13 without regressing the near-zero cross-lingual natural language inference (XNLI) numbers. Two cautions apply. First, shuffling fixes position leakage but should be verified against the distributional metrics, not just top-1 flips, since close-call items can keep their answer while their probabilities swing. Second, any temperature refit should run on permuted data, or the shipped calibration will inherit one arbitrary ordering.

For Maestro Flash, which this graph defines as a proposal with no trained weights and no model results, the implication is process. The proposal already lists option-order flip rate among its target measurements across its evaluation suites, which is the right call: flip rate should be a first-class metric from the first trained checkpoint, not a post hoc analysis. Three specific commitments would strengthen the plan. First, adopt the protocol in this report (seeded permutations, canonical reordering, per-bucket stratification) so Maestro Flash numbers are comparable with the Laya and Jev figures above. Second, treat the decoupled option-encoding hypothesis as an ablation with a flip-rate readout at high cardinality, since that is the regime where the incumbent marker scheme is weakest. Third, fit calibration temperatures on permuted data and report pre- and post-refit flip rates separately, because symmetric rescaling cannot fix order dependence and the two axes must be shown not to trade against each other. The sibling calibration paper’s consistency-penalty proposal is the natural training counterpart to this measurement plan; this report is the harness that would tell whether that penalty works.

One cross-cutting note: open encoder backbones make the encoder-side experiments replicable in a way decoder-judge experiments sometimes are not. A Hugging Face Hub application programming interface (API) search fetched for this report lists openly downloadable ModernBERT checkpoints, the same family Laya builds on, so any lab can rerun the shuffling and invariance ablations above on comparable backbones.

Limitations

The LLM multiple-choice literature studies decoder models predicting option IDs, and its headline mechanisms (token priors) may not transfer to encoder heads that score options without IDs. The transferable parts are the metrics, the uncertainty interaction, and the training-time consistency ideas, not the specific bias magnitudes. The typed-decision numbers themselves are thin: one independent Jev measurement and one self-reported Laya table, each on a handful of suites, with no confidence intervals published in the fetched documents. Jev’s internals are closed, so every causal claim about Jev specifically is behavioral inference, not mechanism. This report ran no new experiments, trained no models, and shipped nothing; every number above is a cited value from a fetched source, and anything not traceable to such a source is marked UNVERIFIED or left out. Finally, all cited figures are snapshots of fast-moving repositories and leaderboards, current as fetched in September 2026, and reruns may legitimately differ.

Open questions

1. Does permutation-group training with a consistency reward cut encoder flip rates without hurting accuracy or calibration?
2. What seeded permutation count, option-count stratification, and distributional readout should a standard typed-decision flip-rate protocol require?
3. At high option cardinality, does architectural invariance beat training-time shuffling on flip rate per unit of latency?

References

1. Pezeshkpour and Hruschka, Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions, arXiv:2308.11483
2. Zheng et al., Large Language Models Are Not Robust Multiple Choice Selectors, arXiv:2309.03882
3. Wang et al., Large Language Models are not Fair Evaluators, arXiv:2305.17926
4. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, arXiv:2306.05685
5. Alzahrani et al., When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards, arXiv:2402.01781
6. Choi et al., Mitigating Selection Bias with Node Pruning and Auxiliary Options, arXiv:2409.18857
7. Guda et al., Quantifying and Mitigating Selection Bias in LLMs, arXiv:2511.21709
8. Zheng et al., Mitigating Selection Bias in Large Language Models via Permutation-Aware GRPO, arXiv:2603.21016
9. Zaheer et al., Deep Sets, arXiv:1703.06114
10. Lee et al., Set Transformer, arXiv:1810.00825
11. Laya BENCHMARKS.md, option-order robustness table
12. nibzard/decision-model-benchmark, independent Jev and LLM measurements
13. Hugging Face model API, ModernBERT search results fetched for this report

← All papers