Constructing Unanswerable Items for Typed-Decision Benchmarks

This report answers one of the open questions from the parent abstention survey: how a typed-decision benchmark should construct unanswerable items so that abstention precision and recall are exactly gradable without a judge. It takes two things from the parent as given and does not re-derive them: the scoring framework (abstention precision and recall on items labeled unanswerable by construction, plus selective risk and coverage) and the schema recommendation (an explicit abstain output with a closed set of reason codes). What follows is measurement design: three construction recipes, a grading protocol, and the controls that keep the labels honest. Velofy has trained no model for this report and ships none with it.

The question

A typed-decision item is a state plus a declared question with a closed output: a label from an option set, a score on a rubric, or a boolean probability. The Laya teardown in this graph documents the three primitives (choice, score, noul) that define the shape. An unanswerable item is one where the correct behavior is to abstain: no output the schema allows is supported by the input. The benchmark’s job is to make that property a fact about the item as constructed, checkable by a script, rather than an opinion about difficulty.

Exact gradability has three requirements. First, the label is fixed at build time by the generator that made the item, not assigned after the fact by a rater or a model. Second, the model’s output is enumerable: an abstain flag plus a reason code from a closed set, so grading is equality on flags, not similarity on prose. Third, the grading rule is public and deterministic: given the item record and the model response, any two graders compute the same score. The parent survey argues a closed schema makes this possible; this report shows what the items must look like for the argument to hold.

The open question names three families. Underspecified state means the input lacks the discriminating fact, so no option is entailed. Contradictory options means the option set defeats a unique correct answer (exact duplicates after normalization), contradicts the state, or rests on a false premise the state denies. Out-of-language input means the state is written in a language outside the model’s declared supported set, so any answer is unvalidated behavior even when it happens to be right. Each family gets its own reason code, its own generator, and its own precision and recall slice, because they fail for different reasons and a single pooled number would hide that.

One design principle runs through all three families: every unanswerable item ships with an answerable twin, the minimal edit that restores answerability (the removed fact put back, the duplicate replaced, the state in a supported language). Twins measure discrimination rather than refusal rate: a model that abstains on everything scores perfect recall and must face its twin accuracy. They also control surface statistics, since twin and item share length, domain, and template, and they give the generator a self-check: the script that removes support must name what it removed. The idea is precedented: MuSiQue adds unanswerable contrast questions to produce MuSiQue-Full [12], FalseQA ships a revised true-premise question alongside each false-premise one [4], and SelfAware pairs unanswerable questions with answerable counterparts [13].

Background

The parent survey (abstention) established the scoring side. Abstention precision is the fraction of refusals that were truly unanswerable; recall is the fraction of unanswerable items refused. Selective risk (error on answered items) and coverage (fraction answered) describe the tradeoff curve. A cost-weighted score combines correct, wrong, and abstain outcomes with application costs. This report takes those definitions as given and asks what the items must guarantee for precision and recall to be computed without human or model judgment.

The closest precedent in reading comprehension is the Stanford Question Answering Dataset (SQuAD) 2.0, which combines the 100,000 questions of SQuAD 1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones, and the official site states the requirement operationally: systems must determine when no answer is supported by the paragraph and abstain [1, 15]. The paper reports that a strong neural system reaching 86% F1 on SQuAD 1.1 achieves only 66% F1 on SQuAD 2.0 [1]. Two properties make SQuAD 2.0 the template: unanswerability is defined by the task setup (the paragraph does not support any span), and grading is exact (span match or abstention), with the official evaluation script published and the test set held hidden [15]. The dataset is mirrored on Hugging Face as rajpurkar/squad_v2 [14]. A typed-decision analogue needs the same two properties with options in place of spans.

The reason a judge must be avoided is visible in the open-ended literature. AbstentionBench evaluates abstention across 20 datasets and 20 frontier large language models (LLMs) [2], and its README documents human-validated judges for scalable evaluation of both abstention and response correctness, with data on Hugging Face at facebook/AbstentionBench [16]. Judges are the right tool for free-text refusals, but they bound the benchmark by the judge’s own uncertainty handling, and they add cost, nondeterminism, and version drift to every run. For typed decisions the judge buys nothing: the output is already a flag and a code, so equality suffices. The benchmark should therefore treat any grading step a script cannot perform as a construction defect, not as a job for a bigger model.

The closest precedent on typed decisions is the independent decision-model benchmark (DMB): five suites and 60 cells with published raw logs, $28.34 in measured spend, and 143 network-free tests [17]. Its forced-uncertainty items show every tested LLM admitting ignorance on 97.3 to 100% of them while Jev admits on 49.7%, with an expected calibration error (ECE) of 0.246 against an LLM range of 0.039 to 0.122 [17]. That is binary admit-or-answer grading on closed APIs, and it proves the coarse version of this report’s thesis. What DMB lacks is family-level construction: its forced-uncertainty slice is one pool, not three families with separate generators and scores.

Two scope boundaries from the parent carry over unchanged. Request-level rejection (over-limit cardinality, malformed input, empty state) is input validation, not uncertainty, and must be scored separately from decision-level abstention. And confidence gating by the caller complements but does not replace an explicit abstain output: grading refusal requires refusal to be in the schema.

What the evidence says

The literature offers at least four mature construction methods for unanswerable items, plus multilingual and contradiction resources that cover the third and second families. The table below compares them on what matters for a typed-decision benchmark: which family each method serves, how it constructs items, how it grades, and what transfers.

Four findings cut across the rows. First, automatic construction is viable. AGent re-matches questions with contexts lacking the answer, reports low error rates, and reports comparable performance to SQuAD 2.0 fine-tuning on multiple EQA benchmarks [7]. Zhu et al. generate unanswerable questions with a pair-to-sequence model and report absolute F1 gains of 1.9 on BERT-base and 1.7 on BERT-large from augmentation [8]. A typed-decision benchmark does not need 50,000 crowdworker items to start; it needs generators with measured, published error rates.

Second, the twin design is precedented in three places. MuSiQue-Full adds unanswerable contrasts to a 25K multihop set and shows single-hop models dropping 30 F1 points, evidence the contrasts defeat shortcuts [12]. FalseQA ships each of its 2,365 false-premise questions with an explanation and a revised true-premise question, and 256 examples suffice for models to discriminate the kind [4]. SelfAware pairs five unanswerable categories with answerable counterparts across 20 tested LLMs [13]. Discrimination between minimally different items is the measured skill, not refusal in isolation.

Third, underspecification deserves the largest slice. AmbigNQ covers 14,042 questions and finds over half of NQ-open ambiguous, with diverse sources such as event and entity references [3]. AbstentionBench’s spectrum includes underspecification alongside unknown answers, false premises, subjectivity, and stale information [2], with 3 new underspecified reasoning challenges among its 20 datasets [16]. If most real questions carry some ambiguity, a benchmark that treats underspecification as a corner case misrepresents deployment.

Fourth, multilingual evaluation already knows how to hold out languages. XNLI extends MultiNLI to 15 languages including Swahili and Urdu, and reports that directly translating test data is the best available baseline [10]. XTREME evaluates 40 languages and 9 tasks and reports a sizable gap between English and transferred performance [9]. The out-of-language family can reuse established language menus and per-language scoring.

Analysis

This section turns the evidence into buildable recipes. All three families share one item record and one grading rule, then diverge in their generators.

The shared item record is a JavaScript Object Notation (JSON) object with these fields: a stable id, the family name, the generator name and version, the random seed, the state, the declared question with its options, the expected output (abstain true plus one reason code), and the twin id pointing at the answerable counterpart. Deterministic generators plus recorded seeds make every item reproducible from the generator version alone. The grading rule is equality on the abstain flag and the reason code: a response is correct on an unanswerable item if and only if it abstains with the family’s code. No partial credit, no paraphrase acceptance, no judge. Per-family abstention precision and recall are computed on the pooled responses, and twin accuracy (exact match on the answerable twins) is reported alongside so that an always-abstain policy is visible as recall 1.0 with twin accuracy at floor. The benchmark fixes and publishes the answerable-to-unanswerable ratio up front, since precision depends on the base rate, and it reports selective risk and coverage over twins plus items using the parent’s definitions (abstention) rather than redefining them here.

Underspecified-state items assert that the state lacks the discriminating fact. Three generators cover the space. Redaction takes an answerable twin and deletes the single supporting sentence, keeping everything else byte-identical; the generator records the deleted span, and a validity script asserts the deleted span was the only support by checking that no other sentence in the state contains the answer string or its listed aliases. Re-matching follows AGent [7]: pair a real question with a same-domain state that lacks the needed information, selected by retrieval similarity so the pair looks topically coherent, with a recorded similarity band to keep difficulty calibrated. Reference ambiguity follows AmbigNQ [3]: construct states containing two or more plausible referents for the question’s key entity or event, so that two options are each supported under a different reading. The reason code is underspecified_state. The twin restores the fact, the original state, or the unique referent. The known failure mode is generator leakage, where redaction leaves detectable scars (length drops, dangling references), so the benchmark must publish length and token-overlap distributions for items versus twins and reject generator versions whose distributions separate trivially.

Contradictory-option items assert that the option set defeats a unique correct answer, and they come in three subforms with one shared code, contradictory_options. Exact duplicates inject a second option identical to an existing one after a published normalization rule (case folding, whitespace collapsing, punctuation stripping); the validity check is string equality after normalization, which a script performs exactly. Paraphrase duplicates are deliberately excluded from version one: near-duplicate detection needs a similarity threshold, and a threshold is a judge by another name. State-contradicting options adapt MultiNLI [11]: take a contradiction-labeled premise-hypothesis pair, use the premise as the state sentence and the hypothesis as an option, so the state denies exactly what the option asserts, with the 433K pairs over 10 genres as source material. False-premise questions adapt FalseQA [4]: the question presupposes something the state denies (the twin is the revised true-premise question, the same pairing FalseQA ships). TruthfulQA’s category list (817 questions over 38 categories [5]) is the menu for covering false-belief varieties rather than piling volume into one kind. The open twist is that duplicate options also interact with option-order sensitivity, which the permutation-sensitivity paper in this graph studies; the benchmark should fix option order per item by seed and report whether shuffling changes refusal behavior.

Out-of-language items assert that the state falls outside the model’s declared supported set. The generator translates answerable twins into held-out languages chosen from the XNLI and XTREME menus (15 and 40 languages respectively [10, 9]), stratified by script family so the slice contains both same-script cases (Spanish for an English-only model, where token statistics barely shift) and different-script cases (Swahili, Urdu, Khmer, where they shift sharply). The label is assigned by the generator’s language tag plus a Unicode script-range check, both deterministic, never by asking a model what language it sees. The reason code is out_of_language, and grading is on the abstention decision alone: a correct label on an out-of-language item still scores wrong, because answering in an undeclared language is unvalidated behavior. The Laya teardown documents why this strictness is load-bearing: confidence saturating near 1.0 where accuracy is zero is exactly what the slice must catch. Scoring runs end to end including any router, since routing is part of the product; a router that sends Khmer to English weights has failed the item even if the weights then answer correctly by luck. The Indic benchmark in this graph is the natural sibling for extending the language menu beyond the XNLI and XTREME lists.

Three controls keep the construction honest. First, adversarial similarity auditing: per generator version, publish probe results showing items and twins are not separable by length, template n-grams, or perplexity bands. SQuAD 2.0 set this bar with crowdworkers writing unanswerable questions to look similar to answerable ones [1]; automatic generators must meet a measured version of it. Second, generator holdout: reserve one generator per family whose items never appear publicly, so training on the benchmark cannot memorize refusal triggers. Versions are published but outputs stay hidden, the same pattern as SQuAD’s hidden test set [15]. Third, human audit of a sampled fraction with a published error rate: AGent’s low reported error rates [7] are the precedent for reporting the number rather than asserting perfection. Audited-answerable items go into a versioned errata file that generator versions reference.

Implications for small decision models

For Jev, the implication is a benchmark-side design, since Jev is a closed application programming interface (API) whose behavior is known only through public materials and third-party measurements, as the Jev survey in this graph documents. The three families grade through whatever refusal surface the API exposes: DMB already demonstrates binary admit-rate scoring on Jev, reporting 49.7% admission against 97.3 to 100% for LLMs [17]. The upgrade this report specifies is family-labeled items with twins, so a future DMB-style run can report whether Jev’s misses concentrate in underspecification, contradiction, or language, and whether its twin accuracy shows the misses are honest uncertainty rather than incapacity. One exclusion matters: Jev’s sharp 255-choice cap, where DMB reports 100% accuracy at 254 and 255 options and a 400 rejection at 256 and above [17], is request-level validation and must stay out of the abstention score, per the parent’s separation rule. Cardinality stress belongs to the option-cardinality design, not to refusal quality.

For Laya, the path is code the project can merge. Laya’s README, fetched for this report, describes typed choice, score, and noul questions in a single forward pass at 33 ms per question and 7.2 ms batched on a T4, with three checkpoints: a 421M ModernBERT-large English model at 512 context, a 322M mmBERT-base multilingual model at 1024 context, and a typed-decisions checkpoint [19]. Its public checkpoints carry text-classification pipeline tags on Hugging Face under convaiinnovations [18]. Three additions would make Laya the reference implementation for this report. First, add the abstain output with the three reason codes to the answer schema, so refusal becomes exactly gradable instead of caller-thresholded. Second, adopt the twin protocol in the project’s own benchmark: every unanswerable item paired, every generator seeded and versioned. Third, use the English versus multilingual checkpoint pair as the built-in out-of-language experiment: the same held-out-language slice should show the English checkpoint abstaining and the multilingual checkpoint answering, which tests both the weights and the router’s checkpoint selection in one run.

For Velofy’s proposed Maestro Flash, the three families belong in the evaluation harness before any weights exist: the item record schema with generator, seed, and twin fields; one implemented generator per family with a published similarity audit; and per-family abstention precision and recall as release gates alongside twin accuracy. The Maestro Flash proposal already plans per-bucket calibration; the abstention analogue is per-family operating points, since answering an out-of-language item need not cost the same as answering an underspecified one. Nothing here claims Maestro Flash has been trained or evaluated. The first numbers should come from baselines run against the harness, not from new weights; harness first with published seeds is what lets later results be trusted.

Limitations

This report ran no experiments; the recipes are synthesized from precedents, and their error rates, difficulty calibration, and audit costs are unmeasured. Excluding paraphrase duplicates from version one is deliberate: exact duplicates grade exactly, near-duplicates do not, and the boundary rule is the first open question. Translation-based out-of-language items risk translationese, artifacts that reveal the slice without revealing the language; human translation of a small seed set plus machine translation for volume is the obvious mitigation, with unmeasured cost. The three-code set is minimal by design, and finer codes multiply the audit burden one code at a time. Human audit is the residual judge: sampling bounds it, but the defended claim is only that grading is judge-free, not construction. Explanation quality (why the model refused, in the FalseQA rebuttal sense [4]) stays judge-graded where needed.

Open questions

1. What paraphrase-similarity rule admits near-duplicate contradictory options while keeping grading exact and judge-free?
2. How small can a human-audited seed set be while holding generator error rate below a stated bound across all three families?
3. Which held-out language set best separates router failure from encoder failure on out-of-language items?

References

1. Rajpurkar et al., Know What You Don’t Know: Unanswerable Questions for SQuAD. arXiv:1806.03822
2. Kirichenko et al., AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions. arXiv:2506.09038
3. Min et al., AmbigQA: Answering Ambiguous Open-domain Questions. arXiv:2004.10645
4. Hu et al., Won’t Get Fooled Again: Answering Questions with False Premises (FalseQA). arXiv:2307.02394
5. Lin et al., TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958
6. UAEval4RAG: Unanswerability Evaluation for Retrieval Augmented Generation. arXiv:2412.12300
7. AGent: A Novel Pipeline for Automatically Creating Unanswerable Questions. arXiv:2309.05103
8. Zhu et al., Learning to Ask Unanswerable Questions for Machine Reading Comprehension. arXiv:1906.06045
9. Hu et al., XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization. arXiv:2003.11080
10. Conneau et al., XNLI: Evaluating Cross-lingual Sentence Representations. arXiv:1809.05053
11. Williams et al., A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (MultiNLI). arXiv:1704.05426
12. Trivedi et al., MuSiQue: Multihop Questions via Single-hop Question Composition. arXiv:2108.00573
13. Yin et al., Do Large Language Models Know What They Don’t Know? (SelfAware). arXiv:2305.18153
14. rajpurkar/squad_v2 dataset on Hugging Face (dataset record via API). https://huggingface.co/datasets/rajpurkar/squad_v2
15. The Stanford Question Answering Dataset official site (SQuAD 2.0 data, evaluation script, license). https://rajpurkar.github.io/SQuAD-explorer/
16. AbstentionBench official repository (scenarios, judges, Hugging Face data). https://github.com/facebookresearch/AbstentionBench
17. nibzard/decision-model-benchmark (forced-uncertainty items, suites, spend, tests). https://github.com/nibzard/decision-model-benchmark
18. convaiinnovations/laya model card on Hugging Face (checkpoint record via API). https://huggingface.co/convaiinnovations/laya
19. NandhaKishorM/laya repository README (primitives, checkpoints, latency figures). https://github.com/NandhaKishorM/laya

← All papers