A Standard Flip-Rate Protocol for Typed Decisions

The question

What seeded permutation count, option-count stratification, and distributional readout should a standard typed-decision flip-rate protocol require? This report answers that question with a concrete specification: an exact number of orders per item, exact bucket boundaries, and an exact readout set, each defended from fetched evidence.

Three things are taken as settled by sibling papers and are not re-argued here. The permutation-sensitivity report establishes flip rate as the primary metric, canonical reordering before comparison, and the need for seeded permutations with per-bucket stratification. The option-cardinality survey establishes that accuracy, expected calibration error (ECE), and flip rate must be reported as curves against option count, not as single numbers. The Jev survey and the Laya teardown supply the reference measurements this protocol must reproduce: Jev at 13% flips in an independent benchmark, Laya at 0.150 and 0.230 on its 20-option suite. This report turns those sketches into a standard two labs can implement independently and compare digit for digit.

First principles

A flip-rate number is built from a few parts. Each part is a choice, and a standard has to fix every one of them.

A flip belongs to an item, not to an answer. The model sees one item under several option orders. Each answer is mapped back to canonical order, and the item either gives the same answer every time or it does not. No gold label enters, so flip rate measures stability, not correctness.

Orders are sampled, not exhausted. An item with n options has n! orders: 24 at 4 options, and far more than any budget allows at 20 or 77. A protocol tests k of them. If an item flips on a fraction q of all orders, k random orders miss the flip with probability (1 minus q)^k. That miss rate sets how small an instability the protocol can see.

Items are sampled too. A bucket of n items gives a rate with a binomial standard error of about the square root of p(1 minus p)/n. More items narrow that error. More orders per item sharpen each item’s verdict but leave the item-sampling error where it is.

Option count changes what is measured. MMLU-Pro, a harder version of the Massive Multitask Language Understanding (MMLU) benchmark, moved from four to ten options and changed both accuracy and prompt sensitivity (details below). A rate pooled across option counts mixes model behavior with suite composition, so each reported number must hold option count roughly fixed.

Top-1 hides the rest of the output. The model returns a probability vector. Flip rate watches only its largest entry, so a second readout must watch the whole vector, or probability mass can move below the top choice unseen.

A result nobody can rerun is not a standard. Unseeded shuffles change with the clock. Seeds, item files and order counts must be published, or two labs cannot compare digit for digit.

Background

Flip-rate numbers today are incomparable, because the three decisions in the title are made differently everywhere or not stated at all. The independent decision-model benchmark (DMB) by nibzard pins its S4 suite to S1 items rerun with permuted option order (3 permutations), with gold labels mapped through the permutation, and it defines flip rate as choice changes under permutation, mapped through the permutation. Laya’s BENCHMARKS.md reports its flip table (0.150, 0.040, 0.000 for the English checkpoint) without stating a permutation count or seed in the fetched text, so its count is UNVERIFIED and no lab can rerun it exactly. In the large language model (LLM) literature, Zheng et al. (arXiv:2309.03882) use cyclic permutation with one ordering per rotation and note that full permutations are prohibitively expensive, while Pezeshkpour and Hruschka (arXiv:2308.11483) test benchmarks with 3, 4, and 5 options per question and report gaps of approximately 13% to 75%. Each of these is a reasonable choice in isolation. Together they produce numbers that cannot be placed side by side.

There are good precedents for what a standard looks like. The EleutherAI lm-evaluation-harness README describes over 60 standard academic benchmarks with the stated rationale that evaluation with publicly available prompts ensures reproducibility and comparability between papers, and its task guide documents a dedicated multiple_choice output type plus group configs under which MMLU averages over per-document accuracies across its 57 subjects. Holistic Evaluation of Language Models (HELM, arXiv:2211.09110) adopts a multi-metric approach, measuring 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency) for each of 16 core scenarios when possible, which its abstract reports as 87.5% of the time. BIG-bench, short for Beyond the Imitation Game (arXiv:2206.04615), reports 204 tasks contributed by 450 authors across 132 institutions, with a human expert rater baseline on every task. The nibzard SPEC adds the sharpest practices for this graph: items frozen at release with item files hashed into the run manifest, S3 regeneration byte-identical given the seed, and S4 holding the prior fixed by reusing the S1 prior so the order experiment isolates order. A flip-rate standard should inherit all four habits: frozen items, hashed manifests, recorded seeds, and one isolated variable.

Two more facts shape the design. First, Python’s own random documentation states that if the seed argument is omitted or None, the current system time is used, which means any protocol that leaves seeding implicit is unreproducible by construction. Second, MMLU-Pro (arXiv:2406.01574) expanded the choice set from four to ten options and reports an accuracy drop of 16% to 33% against MMLU plus a fall in prompt sensitivity from 4-5% to 2% across 24 prompt styles. Option count changes what a benchmark measures. A protocol that pools across option counts will confound model differences with suite composition differences.

What the evidence says

The table below compares the permutation protocols found in fetched sources. The columns are the three standardization decisions plus seeding discipline.

The readout candidates need their own comparison, because the standard must pick one distributional metric to sit beside flip rate.

Three observations from the fetched numbers motivate the analysis section. First, the DMB v2 report shows that top-1 and distributional readouts can disagree sharply: gpt-5.4-mini has the highest S4 flip rate in the table at 37.0% but the lowest mean confidence range at 0.046, while gpt-5.4-nano flips 33.0% with a range of 0.197. A model can swing its answer while its confidence barely moves, or hold its answer while its confidence swings. One readout cannot substitute for the other.

Second, calibration interacts with every choice here. DMB SPEC defines ECE as 10-bin expected calibration error over confidence, and Laya’s BENCHMARKS.md reports that refitting one temperature per (question type, option count) group moves ECE from 0.466 to 0.081 on the English checkpoint. The option-cardinality survey already draws the consequence: calibration parameters must be grouped by option count. The flip-rate protocol inherits that grouping rather than inventing its own.

Third, cost bounds the permutation count from above. DMB reports $28.34 in measured provider spend for its whole grid and $0.07 per 1,000 decisions for Jev on S1, and every added permutation multiplies the billed decisions per item. The LLM literature agrees from the other direction: Zheng et al. call full n! permutation prohibitively expensive, and Guda et al. call naive majority voting computationally prohibitive. The standard must pick the smallest count that still detects flips reliably, not the largest count a budget tolerates.

Analysis

This section makes the three decisions. Each subsection states a requirement, then the evidence and arithmetic behind it.

Decision 1: identity plus 5 seeded permutations (6 orders per item).

The protocol requires, per item: the identity order plus 5 distinct non-identity permutations drawn with a published seed list, for 6 forward passes per item. When fewer than 5 distinct non-identity permutations exist (2 options admit 1, 3 options admit 5, since 3! minus 1 equals 5), the protocol uses all of them. Permutations are drawn per item from a seeded Fisher-Yates shuffle, the seed list is published with the suite, and regeneration from seeds must be byte-identical, following the DMB S3 practice. Gold labels map through the permutation, following the DMB S4 rule, and predicted distributions are un-permuted to canonical option order before any comparison, following the sibling permutation-sensitivity report.

Why 5 and not 3 (DMB) or n (cyclic) or 24 (full factorial at 4 options)? Three arguments converge. First, detection power. Suppose an item flips on a fraction q of all possible orders. With k random non-identity permutations, the chance of missing the flip entirely is (1 minus q)^k. For a coin-flip item (q equals 0.5), k equals 3 misses 12.5% of flippers while k equals 5 misses 3.1%. For a mild item (q equals 0.2), k equals 3 misses 51.2% while k equals 5 misses 32.8%. The step from 3 to 5 buys most of the detection gain available before cost doubles again, and the arithmetic is simple enough to print in the report. Second, cost proportionality. Six forward passes per item doubles the DMB S4 budget rather than multiplying it by n, which cyclic permutation would do (77 passes at Banking77 scale, 255 at the Jev cap). Third, comparability across option counts. A fixed k means the 2-option suite and the 77-option suite get the same measurement effort per item, so cross-bucket differences reflect the model, not the ruler. Cyclic permutation confounds the two by construction.

The edge cases are part of the standard, not footnotes. Binary items contribute only 2 orders; the report must print the realized order count per bucket so a reader can see that the 2-option bucket ran 2 orders per item while the rest ran 6. Ties for top-1 within one order must be broken by a stated rule (lowest canonical index wins) and counted as non-flips only if the tie set is identical across orders; otherwise a model that returns uniform distributions would score a misleading zero. Items where any order fails (refusal, malformed output, the 400 the DMB reports for Jev at 256+ options) are excluded from flip rate with the exclusion count printed, following the DMB rule that malformed and failed decisions count in their own columns, never as wrong answers.

Decision 2: six buckets (2, 3-5, 6-10, 11-20, 21-77, 78-255), reported separately, never pooled.

The protocol requires six option-count buckets: 2, 3-5, 6-10, 11-20, 21-77, and 78-255. Every metric is micro-averaged over items within a bucket (each item one vote, following the harness MMLU group rule), and the report prints all six bucket rows. A single pooled flip rate across buckets is forbidden, because suite composition would then determine the headline number.

The boundaries are inherited, not invented. The first three cut points (2, 3-5, 6-10) match the Laya calibration buckets documented in the Laya teardown (2, 3-5, 6-10, 11+), so flip-rate rows align with the temperature groups a Laya-style project already fits. The 11-20 bucket covers the MMLU-Pro regime (10 options, per its abstract) and the scale where Laya reports its 0.150 and 0.230 intent-suite flips at 20 options. The 21-77 bucket ends at Banking77’s 77 intents, the mid-cardinality probe DMB uses for S1. The 78-255 bucket ends at the 255-choice cap DMB verified for Jev (100% at 254 and 255, refusal at 256 and above), and it is the only bucket where refusals are expected rather than exceptional.

Each bucket needs enough items to make the flip-rate estimate meaningful. DMB targets 300 items per suite (200 eval plus 100 for order-stability reruns). This protocol requires at least 100 items per bucket, with the same items reused across the 6 orders, and it requires the prior to be held fixed across orders within an item, following the DMB rule that S4 uses the S1 prior. A 100-item bucket with a true flip rate of 0.13 (the DMB Jev figure) has a binomial standard error of about 0.034, which is coarse but honest; suites that want tighter numbers add items, not permutations, because items are usually the cheaper margin for models billed per application programming interface (API) call and always the more informative one. The report must print n per bucket and must not round away the standard error.

Decision 3: flip rate plus mean pairwise total variation distance plus permuted accuracy and ECE.

The protocol requires four readouts per bucket: (1) flip rate, the fraction of items whose top-1 answer differs across the 6 canonical-order distributions; (2) mean pairwise total variation distance (TVD), the average over all 15 order pairs of half the L1 distance between the two canonical-order distributions; (3) accuracy on the permuted orders against mapped gold labels; (4) 10-bin ECE over reported confidence on the permuted orders, following the DMB SPEC definition.

TVD is the distributional readout because it satisfies every standardization criterion while its rivals each fail one. It is label-free (unlike accuracy-conditioned metrics), symmetric (unlike single-direction divergences), bounded in [0, 1] with 0 if and only if the distributions are identical, parameter-free, and implementable in two lines of NumPy. The DMB confidence-range precedent is coarser: the gpt-5.4-mini paradox above (37.0% flips with 0.046 range) shows that top-1 confidence range can sit near zero while answers swing, because range watches one slot and TVD watches the whole vector. CKLD and PBM are serious candidates with the right shape (both target cross-ordering inconsistency without labels, per their abstracts), but their exact constructions were not verified from fetched sources in this round, and a standard cannot require a formula it cannot print. A future revision should specify one of them once its definition is fetched and pinned; until then TVD is the conservative choice that any lab can implement from this paragraph.

Accuracy and ECE on permuted orders stay in the set because they separate failure modes the first two readouts conflate. A model can be wrong-but-stable (low flip rate, low accuracy) or right-but-unstable (high flip rate, high mean accuracy). DMB already keeps this separation by design: its S4 column reports permuted accuracy (Jev 76.7% in the README table) alongside flip rate (13.0% in v2). ECE on permuted orders catches the case the permutation-sensitivity report flags, where temperatures fit on unpermuted data bake one arbitrary ordering into shipped confidence. If permuted ECE exceeds unpermuted ECE by more than the bucket standard error, the calibration was fit on the wrong distribution.

Implications for small decision models

For Jev, the implication is a rerun at the new count with published seeds. DMB already ran the closest existing measurement (3 permutations, Jev at 13.0% flips, 0.092 mean confidence range, 76.7% permuted accuracy), but S4 covers only the 77-option point, and the fetched text does not publish S4 seeds. A team depending on Jev should rerun the six-bucket suite at 6 orders per item on its own label sets, because every stratified source in this graph shows flip behavior varying by suite and option count, and because Jev’s flat 264-276 ms median (p50) latency from 2 to 255 options (per DMB) means the rerun cost is predictable: six times the single-order bill, with no latency surprise per order. The 78-255 bucket doubles as a cap probe, since DMB verified refusal at 256+. Nothing here requires TypeSafe’s cooperation; it is all client-side measurement.

For Laya, the implication is to publish what the standard requires and rerun what it cannot publish retroactively. Concretely: state the permutation count and seeds behind the current 0.150, 0.040, 0.000 table (UNVERIFIED as fetched), or rerun those three suites at the 6-order rule; extend the suite set so all six buckets are covered (the current three suites leave 2-option, mid-cardinality non-intent, and 78+ behavior unmeasured); report TVD beside flip rate, since Laya’s marker-per-option head (per the Laya teardown) can plausibly shift sub-top-1 mass under permutation even when top-1 holds; and refit the per-(type, option-count) temperatures on permuted orders, because the current refit (ECE 0.466 to 0.081 on English, per BENCHMARKS.md) ran on unpermuted data as far as fetched sources show. Because Laya is open source with openly downloadable ModernBERT-family backbones (the Hugging Face API reports 4,372,504 downloads for answerdotai/ModernBERT-base and 625,057 for answerdotai/ModernBERT-large at fetch time), every one of these steps is replicable by any lab, which makes Laya the natural first adopter of the standard.

For Velofy’s proposed Maestro Flash, which has no weights and no results, the implication is process, and the proposal already points the right way: its evaluation plan lists option-order flip rate among its target measurements across MASSIVE, XNLI, Banking77, AG News, DAIR Emotion, workflow, and Indic suites (per the Maestro Flash proposal). Three commitments would make that plan standard-conformant from the first checkpoint. First, freeze the six-bucket item files with hashes in the run manifest before the first training run, following DMB practice, so later suite edits cannot move the goalposts. Second, log all four readouts (flip rate, TVD, permuted accuracy, permuted ECE) per bucket from the first evaluation, not as a post hoc analysis, and gate release notes on the 21-77 and 78-255 buckets where the incumbent marker scheme is weakest. Third, treat the decoupled option-encoding hypothesis as an ablation scored by this protocol, with TVD as the tiebreaker if two variants match on flip rate. The sibling calibration proposal supplies the natural training counterpart (a consistency penalty across permuted group members); this protocol is the harness that would tell whether that penalty works.

Limitations

This report ran no experiments, trained no models, and shipped nothing; every number is a cited value from a fetched source, and anything not traceable to such a source is marked UNVERIFIED or left out. The 6-order recommendation rests on detection-power arithmetic and cost proportionality, not on a measured stability curve, because no fetched source reruns the same suite at multiple permutation counts with disjoint seeds. The bucket boundaries are inherited from Laya buckets, MMLU-Pro, Banking77, and the Jev cap, which makes them principled but also contingent: a lab whose label sets cluster at 30 options will find the 21-77 bucket too coarse and should subdivide it while keeping the standard rows for comparability. TVD is a conservative placeholder, not a proven optimum; if a future round pins the CKLD or PBM formula from primary text, the standard should adopt whichever predicts downstream harm better. The LLM-side evidence (cyclic permutation, CKLD, PBM, MMLU-Pro) studies decoder models predicting option IDs, and only its protocol shapes transfer to encoder heads, not its magnitudes. Finally, all repository figures are snapshots as fetched in September 2026; DMB and Laya are fast-moving projects, and reruns may legitimately differ.

Open questions

1. What permutation count makes flip-rate estimates stable across seeds, measured by rerunning the 6-order protocol with disjoint seed sets and reporting rank agreement across models?
2. At which option counts do flip-rate and TVD curves actually bend on fixed item sets, and do the proposed bucket boundaries match those bends or need reslicing?
3. Does mean pairwise TVD predict selective-automation loss (accuracy at fixed coverage) better than top-1 flip rate on held-out suites?

References

1. Zheng et al., Large Language Models Are Not Robust Multiple Choice Selectors, arXiv:2309.03882
2. Pezeshkpour and Hruschka, Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions, arXiv:2308.11483
3. Choi et al., Mitigating Selection Bias with Node Pruning and Auxiliary Options, arXiv:2409.18857
4. Guda et al., Quantifying and Mitigating Selection Bias in LLMs, arXiv:2511.21709
5. Du et al., Strengthened Symbol Binding Makes Large Language Models Reliable Multiple-Choice Selectors, arXiv:2406.01026
6. Addressing Blind Guessing: Calibration of Selection Bias in Multiple-Choice QA by Video Language Models, arXiv:2410.14248
7. Hendrycks et al., Measuring Massive Multitask Language Understanding, arXiv:2009.03300
8. Wang et al., MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark, arXiv:2406.01574
9. Guo et al., On Calibration of Modern Neural Networks, arXiv:1706.04599
10. Liang et al., Holistic Evaluation of Language Models, arXiv:2211.09110
11. Srivastava et al., Beyond the Imitation Game, arXiv:2206.04615
12. EleutherAI, lm-evaluation-harness README
13. EleutherAI, lm-evaluation-harness task guide
14. nibzard/decision-model-benchmark SPEC.md
15. nibzard/decision-model-benchmark results v2 report
16. Laya BENCHMARKS.md, NandhaKishorM/laya
17. Hugging Face model API, ModernBERT search results fetched for this report
18. Python documentation, random.seed

← All papers