SearchBook a call

RESEARCH / SMALL MODELS

A graph of
small decision models.

Every paper here ends with open questions. Those questions become the next nodes, and the next papers. Some of them are answered by measurement, not reading.

Papers
38
Measured runs
04
Open questions
70
Rounds
10

Start with the measurements

Results from Velofy's local Decision Bench harness. Every table traces to a raw results file, and every deviation from the protocol is listed.

MEASURED

GLM-5.3 and open-weight cyber capability: what the evidence supports

A source-by-source reading of Anthropic's GLM-5.3 cyber report: what three parties agree on, where the framing outruns the data, and what open weights change.

MEASURED

Decision Bench v0, first run: three systems on Banking77

First measured results: a TF-IDF baseline, Laya zero-shot, and GLiClass-small zero-shot on 77-way Banking77, scored on accuracy, calibration, flips, and cost.

MEASURED

Decision Bench experiments: temperature refit and the option-count sweep

Two follow-ups: a train-only temperature refit fixes a trained baseline's calibration, and zero-shot models hold at 5 options but degrade sharply by 77.

MEASURED

Decision Bench experiments: near-miss distractors, per-bucket temperature, and shortlists

Similar distractors erase most of Laya's small-option edge, per-bucket temperature beats one global value, and a shortlist cuts order flips from 55% to 9%.

The map

Columns are rounds. Lines connect a paper to the papers that grew from its open questions. The number beside a node counts the questions it left open.

ROUND 0ROUND 1ROUND 2ROUND 3ROUND 4ROUND 5ROUND 6ROUND 7ROUND 8ROUND 9Typed decisions without generation: What does it take to answer a typed question in one forward pass?Typed decisions withoutgenerationTypeSafe Jev: What is publicly known about Jev and how does it behave under independent tests?TypeSafe JevLaya: How does an open typed-decision engine work, and where does it break?LayaMaestro Flash: What should velofy build as its own small decision model?Maestro FlashProper-scoring-rule RL: Can RL against proper scoring rules make small encoders calibrated?Proper-scoring-rule RLIndic typed-decision benchmark: What would an Indic-first decision benchmark need to measure?Indic typed-decisionbenchmarkEncoder backbones in 2026: Which encoder backbone should a small decision model start from, and why?Encoder backbones in2026Option cardinality limits: Why do decision models break past 20 or 255 options, and what architectures avoid it?Option cardinalitylimits1 openOption-order sensitivity: Why do typed-decision models flip answers when options are reordered, and how is that measured and fixed?Option-order sensitivity1 openAbstention and honest uncertainty: How should a small model say "I do not know", and how do we score it?Abstention and honestuncertaintySub-1B decoders as decision models: How do small decoders with constrained decoding compare with encoders on typed decisions?Sub-1B decoders asdecision models3 openDistilling LLM decisions into encoders: How do we transfer decisions and calibrated probabilities from large models into a small encoder?Distilling LLM decisionsinto encoders1 openGLiNER and GLiClass: What does the GLiNER family teach about zero-shot label scoring?GLiNER and GLiClass1 openDecision economics: What does a million decisions cost across APIs, small encoders, and local hardware?Decision economics1 openStage-zero backbone harness: Which backbone wins a fixed stage-zero decision harness that scores Banking77 high-cardinality choice, ordinal error, Brier score, ECE, p50 latency, and option-order flip rate across ModernBERT-large, mmBERT-base, Ettin-encoder-400M, and DeBERTa-v3-large?Stage-zero backboneharness2 openUsable decision context: How much usable decision context can a Laya-style single-pass head actually exploit on ModernBERT-class backbones before option text crowds out state, and does extending truncation from 1024 toward the reported 8192 native length improve high-cardinality accuracy without harming calibration?Usable decision context3 openCost of multilinguality: What is the measured cost of multilinguality on native-text Indic decision benchmarks for mmBERT-base against an English ModernBERT control behind a router, scored per language rather than averaged?Cost of multilinguality3 openCalibration grouping by option count: How should temperature or calibration parameters be grouped by option count so that ECE and flip rate stay flat as label sets grow.Calibration grouping byoption count3 openShortlist rerank protocol: What shortlist then rerank protocol preserves Banking77 accuracy and calibration past single pass limits while reporting shortlist recall separately.Shortlist rerankprotocol3 openPermutation-group training for small encoders: Does permutation-group training with a consistency reward cut encoder flip rates without hurting accuracy or calibration?Permutation-grouptraining for small…3 openA standard flip-rate protocol: What seeded permutation count, option-count stratification, and distributional readout should a standard typed-decision flip-rate protocol require?A standard flip-rateprotocol3 openCheapest trigger beating max-softmax: What is the cheapest abstention trigger that beats max-softmax thresholding on a small typed-decision encoder, and on which slices does it win?Cheapest trigger beatingmax-softmax3 openConstructing gradable unanswerable items: How should a typed-decision benchmark construct unanswerable items (underspecified state, contradictory options, out-of-language input) so that abstention precision and recall are exactly gradable without a judge?Constructing gradableunanswerable items3 openWhere refusal should be learned: At what point in the training stack should refusal be learned (pretraining signal, refusal-aware tuning data, selection head, post-hoc threshold), and how do those choices interact with proper-scoring-rule calibration?Where refusal should belearned2 openTeacher temperature and KL direction: What teacher temperature and KL direction for soft-target distillation produce the best post-refit Brier score and ECE on high-cardinality choice heads at matched accuracy.Teacher temperature andKL direction1 openRationale supervision versus soft targets: Does rationale-derived supervision through filtering, loss weighting, or an auxiliary head beat plain soft-target distillation for single-pass encoder decision heads at a matched teacher-query budget.Rationale supervisionversus soft targets3 openCost-per-accepted-decision benchmark design: What would a cost-per-accepted-decision benchmark need to fix (prompt shapes, token accounting, error budgets, serving utilization) so that one dollars-per-million figure is comparable across APIs, rented GPUs, and local hardware?Cost-per-accepted-decisionbenchmark design3 openSelf-host crossover throughput: At what sustained decisions-per-second does a Maestro-class open encoder on a current budget GPU undercut the cheapest measured API on identical prompts, and how do cold starts and batching delays move that crossover?Self-host crossoverthroughput3 openCalibrating sigmoid label scorers: What is the cheapest calibration recipe that gives a GLiNER-style per-label sigmoid scorer usable selective-automation curves: does per-option-count temperature fitting suffice, or does the independent-versus-softmax structure of the scores require a proper-scoring-rule training change?Calibrating sigmoidlabel scorers3 openExamples per label on fine-grained intents: How many examples per label does a small decision encoder need on fine-grained intent sets: can the GLiClass 8-shot recovery pattern be replicated on Banking77-scale tasks with frozen encoders and example-augmented label representations, or does it require fine-tuning?Examples per label onfine-grained intents3 openCardinality break point: At what option cardinality does each trunk's accuracy break under the single-pass head, and does the break point track native context length (8192 for ModernBERT-large, 512 for DeBERTa-v3-large, UNVERIFIED for the others) or a head-design ceiling shared by all four?Cardinality break point3 openTeacher calibration before distillation: Does calibrating the teacher with temperature scaling or LoCa-style correction before distillation improve the student's post-refit ECE, or only its accuracy.Teacher calibrationbefore distillation3 openSymmetric KL and tail dark knowledge: Does symmetric or reverse KL preserve non-target dark knowledge on high-cardinality heads better than forward KL after the calibration refit, or does the refit erase the difference.Symmetric KL and taildark knowledge3 openVelofy Decision Bench v0: What single, frozen benchmark protocol should measure accuracy, calibration, abstention, option-order flips, cardinality, and cost per accepted decision for small decision models?Velofy Decision Bench v03 openThe smallest selection head that wins: What is the smallest selection-head or abstain-class design that beats per-bucket post-hoc thresholding on risk-coverage for a small typed-decision encoder?The smallest selectionhead that wins3 openDecision Bench v0, first run: What happens when a trained baseline, Laya, and GLiClass are actually measured on one 77-way task?Decision Bench v0, firstrun1 openDoes a refit fix the baseline?: Does a post-hoc temperature refit on held-out permuted data close the TF-IDF baseline's calibration gap without hurting selective coverage?Does a refit fix thebaseline?Flips and coverage as options grow: How do flip rate and selective coverage change for each system as the option set grows from 5 to 20 to 77 on the same items?Flips and coverage asoptions growRefit and option-count sweep: Does a temperature refit fix calibration, and how do zero-shot decision models change from 5 to 77 options?Refit and option-countsweepNear-miss distractors: With distractors chosen as the nearest labels in embedding space, how much of Laya's small-k advantage survives?Near-miss distractorsPer-bucket temperature for zero-shot models: Does one temperature per option-count bucket recover selective coverage for zero-shot models at 20 options?Per-bucket temperaturefor zero-shot modelsShortlist, then decide: At what option count does a trained top-5 shortlist followed by Laya beat either system alone on accuracy and flip rate?Shortlist, then decideNear-miss, per-bucket temperature, shortlists: Do near-miss distractors, per-bucket temperature, and a trained shortlist change what zero-shot decision models can do?Near-miss, per-buckettemperature, shortlists3 open

Papers by round

ROUND 0

  1. Typed decisions without generationWhat does it take to answer a typed question in one forward pass?
  2. TypeSafe JevWhat is publicly known about Jev and how does it behave under independent tests?from Typed decisions without generation
  3. LayaHow does an open typed-decision engine work, and where does it break?from Typed decisions without generation
  4. Maestro FlashWhat should velofy build as its own small decision model?from TypeSafe Jev, Laya
  5. Proper-scoring-rule RLCan RL against proper scoring rules make small encoders calibrated?from Laya
  6. Indic typed-decision benchmarkWhat would an Indic-first decision benchmark need to measure?from Maestro Flash

ROUND 1

  1. Encoder backbones in 2026Which encoder backbone should a small decision model start from, and why?from Maestro Flash
  2. Option cardinality limitsWhy do decision models break past 20 or 255 options, and what architectures avoid it?from TypeSafe Jev, Laya
  3. Option-order sensitivityWhy do typed-decision models flip answers when options are reordered, and how is that measured and fixed?from TypeSafe Jev
  4. Abstention and honest uncertaintyHow should a small model say "I do not know", and how do we score it?from TypeSafe Jev, Proper-scoring-rule RL

ROUND 2

  1. Sub-1B decoders as decision modelsHow do small decoders with constrained decoding compare with encoders on typed decisions?from Typed decisions without generation
  2. Distilling LLM decisions into encodersHow do we transfer decisions and calibrated probabilities from large models into a small encoder?from Maestro Flash
  3. GLiNER and GLiClassWhat does the GLiNER family teach about zero-shot label scoring?from Typed decisions without generation
  4. Decision economicsWhat does a million decisions cost across APIs, small encoders, and local hardware?from TypeSafe Jev

ROUND 3

  1. Stage-zero backbone harnessWhich backbone wins a fixed stage-zero decision harness that scores Banking77 high-cardinality choice, ordinal error, Brier score, ECE, p50 latency, and option-order flip rate across ModernBERT-large, mmBERT-base, Ettin-encoder-400M, and DeBERTa-v3-large?from Encoder backbones in 2026
  2. A standard flip-rate protocolWhat seeded permutation count, option-count stratification, and distributional readout should a standard typed-decision flip-rate protocol require?from Option-order sensitivity
  3. Constructing gradable unanswerable itemsHow should a typed-decision benchmark construct unanswerable items (underspecified state, contradictory options, out-of-language input) so that abstention precision and recall are exactly gradable without a judge?from Abstention and honest uncertainty
  4. Teacher temperature and KL directionWhat teacher temperature and KL direction for soft-target distillation produce the best post-refit Brier score and ECE on high-cardinality choice heads at matched accuracy.from Distilling LLM decisions into encoders
  5. Cost-per-accepted-decision benchmark designWhat would a cost-per-accepted-decision benchmark need to fix (prompt shapes, token accounting, error budgets, serving utilization) so that one dollars-per-million figure is comparable across APIs, rented GPUs, and local hardware?from Decision economics
  6. Calibrating sigmoid label scorersWhat is the cheapest calibration recipe that gives a GLiNER-style per-label sigmoid scorer usable selective-automation curves: does per-option-count temperature fitting suffice, or does the independent-versus-softmax structure of the scores require a proper-scoring-rule training change?from GLiNER and GLiClass

ROUND 4

  1. Cost of multilingualityWhat is the measured cost of multilinguality on native-text Indic decision benchmarks for mmBERT-base against an English ModernBERT control behind a router, scored per language rather than averaged?from Encoder backbones in 2026
  2. Permutation-group training for small encodersDoes permutation-group training with a consistency reward cut encoder flip rates without hurting accuracy or calibration?from Option-order sensitivity
  3. Where refusal should be learnedAt what point in the training stack should refusal be learned (pretraining signal, refusal-aware tuning data, selection head, post-hoc threshold), and how do those choices interact with proper-scoring-rule calibration?from Abstention and honest uncertainty
  4. Cardinality break pointAt what option cardinality does each trunk's accuracy break under the single-pass head, and does the break point track native context length (8192 for ModernBERT-large, 512 for DeBERTa-v3-large, UNVERIFIED for the others) or a head-design ceiling shared by all four?from Stage-zero backbone harness
  5. Teacher calibration before distillationDoes calibrating the teacher with temperature scaling or LoCa-style correction before distillation improve the student's post-refit ECE, or only its accuracy.from Teacher temperature and KL direction
  6. Velofy Decision Bench v0What single, frozen benchmark protocol should measure accuracy, calibration, abstention, option-order flips, cardinality, and cost per accepted decision for small decision models?from A standard flip-rate protocol, Constructing gradable unanswerable items, Cost-per-accepted-decision benchmark design, Stage-zero backbone harness

ROUND 5

  1. Rationale supervision versus soft targetsDoes rationale-derived supervision through filtering, loss weighting, or an auxiliary head beat plain soft-target distillation for single-pass encoder decision heads at a matched teacher-query budget.from Distilling LLM decisions into encoders
  2. Self-host crossover throughputAt what sustained decisions-per-second does a Maestro-class open encoder on a current budget GPU undercut the cheapest measured API on identical prompts, and how do cold starts and batching delays move that crossover?from Decision economics

ROUND 6

  1. Shortlist rerank protocolWhat shortlist then rerank protocol preserves Banking77 accuracy and calibration past single pass limits while reporting shortlist recall separately.from Option cardinality limits
  2. Examples per label on fine-grained intentsHow many examples per label does a small decision encoder need on fine-grained intent sets: can the GLiClass 8-shot recovery pattern be replicated on Banking77-scale tasks with frozen encoders and example-augmented label representations, or does it require fine-tuning?from GLiNER and GLiClass

ROUND 7

  1. Cheapest trigger beating max-softmaxWhat is the cheapest abstention trigger that beats max-softmax thresholding on a small typed-decision encoder, and on which slices does it win?from Abstention and honest uncertainty
  2. The smallest selection head that winsWhat is the smallest selection-head or abstain-class design that beats per-bucket post-hoc thresholding on risk-coverage for a small typed-decision encoder?from Where refusal should be learned
  3. Decision Bench v0, first runWhat happens when a trained baseline, Laya, and GLiClass are actually measured on one 77-way task?from Velofy Decision Bench v0

ROUND 8

  1. Does a refit fix the baseline?Does a post-hoc temperature refit on held-out permuted data close the TF-IDF baseline's calibration gap without hurting selective coverage?from Decision Bench v0, first run
  2. Flips and coverage as options growHow do flip rate and selective coverage change for each system as the option set grows from 5 to 20 to 77 on the same items?from Decision Bench v0, first run
  3. Refit and option-count sweepDoes a temperature refit fix calibration, and how do zero-shot decision models change from 5 to 77 options?from Decision Bench v0, first run, Does a refit fix the baseline?, Flips and coverage as options grow

ROUND 9

  1. Usable decision contextHow much usable decision context can a Laya-style single-pass head actually exploit on ModernBERT-class backbones before option text crowds out state, and does extending truncation from 1024 toward the reported 8192 native length improve high-cardinality accuracy without harming calibration?from Encoder backbones in 2026
  2. Calibration grouping by option countHow should temperature or calibration parameters be grouped by option count so that ECE and flip rate stay flat as label sets grow.from Option cardinality limits
  3. Symmetric KL and tail dark knowledgeDoes symmetric or reverse KL preserve non-target dark knowledge on high-cardinality heads better than forward KL after the calibration refit, or does the refit erase the difference.from Teacher temperature and KL direction
  4. Near-miss distractorsWith distractors chosen as the nearest labels in embedding space, how much of Laya's small-k advantage survives?from Refit and option-count sweep
  5. Per-bucket temperature for zero-shot modelsDoes one temperature per option-count bucket recover selective coverage for zero-shot models at 20 options?from Refit and option-count sweep
  6. Shortlist, then decideAt what option count does a trained top-5 shortlist followed by Laya beat either system alone on accuracy and flip rate?from Refit and option-count sweep
  7. Near-miss, per-bucket temperature, shortlistsDo near-miss distractors, per-bucket temperature, and a trained shortlist change what zero-shot decision models can do?from Refit and option-count sweep

Open questions

Grouped by the paper that raised them. The next rounds are chosen from here.

Option cardinality limits1 open
  • Decoupled versus joint crossoverAt what option count does decoupled option encoding surpass joint encoding on Banking77 when backbone, training data, and calibration budget are matched.
Option-order sensitivity1 open
  • Invariance by architecture or by augmentationAt high option cardinality, does architectural invariance beat training-time shuffling on flip rate per unit of latency?
Sub-1B decoders as decision models3 open
  • Sub-1B guided-choice head-to-headWhat does a named sub-1B instruct decoder score, under guided choice on fixed hardware, against a matched-size encoder on a shared typed-decision suite measuring accuracy, ECE, flip rate, and p50 latency?
  • Temperature scaling for guided choiceDoes single-parameter temperature scaling calibrate guided-choice decoder scores at sub-1B size, and how does the resulting ECE compare with encoder calibration on the same suite?
  • Decoder advantage crossoverOn the conditions hypothesized to favor decoders (many options, multilingual input, JSON output), does a small decoder actually beat an encoder head, and at what output length does the token-loop latency cost erase the advantage?
Distilling LLM decisions into encoders1 open
  • Calibration refit data versus option countHow much labeled held-out data does the calibration refit need as option count grows, and does per-bucket temperature fitting stay sufficient or require per-language or per-domain grouping.
GLiNER and GLiClass1 open
  • Decoupled versus joint crossoverAt what option count does decoupled option encoding surpass joint encoding on Banking77 when backbone, training data, and calibration budget are matched.
Decision economics1 open
  • Cost curve past 255 optionsHow should the cost curve past 255 options be measured and minimized: shortlist-then-decide pipelines, decoupled option encoders, or hierarchical decisions, scored on dollars per accepted decision rather than accuracy alone?
Stage-zero backbone harness2 open
  • Measured six-axis resultsWhat are the measured six-axis results when the fixed stage-zero harness is actually run, with paired bootstrap intervals, across ModernBERT-large, mmBERT-base, Ettin-encoder-400M, and DeBERTa-v3-large under one head implementation and one training budget?
  • Shuffling vs pretraining stabilityHow much of each trunk's option-order flip rate is removed by the shuffling schedule alone, measured by the disabled-shuffling diagnostic run, and does any backbone show a pretraining-level stability advantage that augmentation cannot explain?
Usable decision context3 open
  • Measured usable context lengthWhat is the measured usable context of a Laya-style single-pass head on ModernBERT-large: at what max_len between 512 and 8192 does Banking77 accuracy per state-length band flatten, and does post-refit ECE stay flat or degrade at each step?
  • Label-length control separationDoes a label-length control separate token starvation from marker-count ceilings: do long paraphrased wordings recover with head-budget relief while the option-count break point stays fixed, or do both move together?
  • Shortlist crossover versus truncationAt what option count and state length does a top-k shortlist followed by one forward pass beat full-set single-pass scoring on accuracy per millisecond at matched post-refit ECE, and does that crossover move as truncation extends?
Cost of multilinguality3 open
  • Per-language native-stratum costWhat is the per-language multilinguality cost of mmBERT-base against an English ModernBERT control behind a router on the native stratum (IndicQA, Naamapadam, and decision-shaped native suites as they appear), scored separately for accuracy, Brier score, ECE, p50 latency, and option-order flip rate?
  • Translationese versus trunk gapHow much of any measured gap is translationese rather than trunk quality, estimated by scoring the same decision tasks in native, professionally localized, and machine-translated forms and reporting the provenance spread alongside the trunk spread?
  • Router misrouting crossoverAt what router misrouting rate does the routed system's total cost (trunk gap on routed traffic plus misrouting errors) exceed the single-trunk alternative, and does that crossover differ between high-resource Indic languages and low-resource ones?
Calibration grouping by option count3 open
  • Per-count ECE bend boundariesAt which option counts do per count ECE curves bend on fixed items, and which bucket boundaries keep residual ECE flat with the fewest held out samples per bucket.
  • Permuted fit versus canonicalDoes fitting one temperature per option count bucket on permuted held out data change per bucket ECE against canonical order fitting, and can any scalar grouping move flip rate or only ECE.
  • Dirichlet versus temperature per bucketDoes per bucket Dirichlet calibration beat per bucket temperature scaling on classwise ECE and Brier at high option counts, at matched held out budget with the clamp and fallback applied.
Shortlist rerank protocol3 open
  • Recall knee versus conditional peakAt what k does frozen bi-encoder shortlist recall saturate on full 77 way Banking77, and does rerank conditional accuracy peak at the same k or at a smaller one.
  • Conditional versus end-to-end calibrationHow should calibration be reported and fitted when the shortlist truncates the distribution: is per k temperature refit on the conditional distribution sufficient, or does end to end selective automation need retriever margin interpolation or an explicit abstention rule on low shortlist margin.
  • Shortlist flip rate versus single passDoes reranking a shortlist reduce option order flip rate versus single pass full set scoring at matched end to end accuracy, and how many seeded permutations per item make that comparison stable across k.
Permutation-group training for small encoders3 open
  • Lambda and group size tradeoffsWhat lambda schedule and group size maximize encoder flip-rate reduction per point of accuracy and ECE cost?
  • Temperature refit after consistency tuningDoes refitting temperatures on permuted data after consistency tuning fully absorb the calibration shift, or does a residual ECE change persist?
  • Consistency term versus augmentation at matched FLOPsAt matched training FLOPs, does the consistency term beat plain augmentation on flip rate for marker-per-option heads?
A standard flip-rate protocol3 open
  • Permutation count stabilityWhat permutation count makes flip-rate estimates stable across seeds, measured by rerunning the 6-order protocol with disjoint seed sets and reporting rank agreement across models?
  • Bucket boundary empiricsAt which option counts do flip-rate and TVD curves actually bend on fixed item sets, and do the proposed bucket boundaries match those bends or need reslicing?
  • TVD versus flip rate predictionDoes mean pairwise TVD predict selective-automation loss (accuracy at fixed coverage) better than top-1 flip rate on held-out suites?
Cheapest trigger beating max-softmax3 open
  • Logit trigger slice showdownWhich logit-only trigger wins on which slice for a small typed-decision encoder: a head-to-head of max-softmax, max logit, margin, entropy, energy, and DOCTOR-style statistics across out-of-scope, near-OOD, high-cardinality, and saturated-confidence slices?
  • Transferable abstention thresholdsHow should per-bucket abstention thresholds be fitted so they transfer across option counts, languages, and backbones without refitting on every deployment's held-out data?
  • Cheapest input-side backstopWhat is the cheapest input-side signal that catches saturated-confidence failures where all output-space triggers agree and are wrong, and how should it combine with logit triggers in one abstention policy?
Constructing gradable unanswerable items3 open
  • Paraphrase rule for near-duplicate optionsWhat paraphrase-similarity rule admits near-duplicate contradictory options while keeping grading exact and judge-free?
  • Minimum audited seed set sizeHow small can a human-audited seed set be while holding generator error rate below a stated bound across all three families?
  • Held-out languages for router vs encoderWhich held-out language set best separates router failure from encoder failure on out-of-language items?
Where refusal should be learned2 open
  • Refusal tuning versus post-hoc selectionDoes refusal-aware tuning data preserve, improve, or degrade selective calibration at fixed coverage compared with a post-hoc-only baseline on the same encoder?
  • Reporting selective calibrationHow should selective calibration be reported alongside refusal metrics so that changing application costs does not require rerunning the evaluation?
Teacher temperature and KL direction1 open
  • Temperature by direction ablation at 77 optionsWhat teacher temperature and KL direction minimize post-refit Brier score and ECE on a 77-way choice head at matched accuracy when the teacher is a calibrated LLM and the student refit is per-bucket.
Rationale supervision versus soft targets3 open
  • Agreement filtering versus soft targets at fixed spendAt a fixed teacher-query spend measured in input plus output tokens, does agreement-filtered teacher supervision beat plain soft-target distillation on a larger unfiltered set for a single-pass choice head, judged on accuracy and post-refit ECE.
  • Auxiliary rationale head and representationDoes an auxiliary rationale-prediction head discarded at inference improve decision accuracy or post-refit calibration over soft targets alone at matched data, and which rationale representation transfers best.
  • Consistency weights and calibration transferDo rationale-consistency loss weights transfer teacher calibration or only accuracy, and does a per-bucket temperature refit erase any difference between weighted and unweighted students.
Cost-per-accepted-decision benchmark design3 open
  • Reference prompt suite freezeWhat reference prompt suite, with fixed option-count strata, state-length bands, pinned revisions, and item hashes, should the benchmark freeze so that tokens per decision stay stable enough to compare a new contender against published dollars-per-million figures without rerunning every baseline?
  • Pricing the over-cap regionHow should the benchmark price the over-cap region past single-pass and vendor choice limits so that shortlist-then-decide pipelines, decoupled option encoders, and hierarchical decisions report one comparable accepted-decision figure?
  • Load-scenario utilization sweepWhat load-scenario trio and utilization sweep should the benchmark require so that rented-GPU dollars per decision at stated latency bounds compares fairly against API dollars per decision at stated tiers, including cold starts and batching delay?
Self-host crossover throughput3 open
  • Gap histogram and scaledown sweepWhat gap histogram and scaledown sweep should a crossover benchmark require so that one sustained dps figure prices reserved, keep warm, and scale to zero pools on the same arrival trace?
  • Timeout batching under latency boundWhat timeout batching policy maximizes decisions per dollar under a 300 ms p50 bound for a Maestro-class encoder on a budget GPU, and how should its added gathering delay be reported separately from service time?
  • Identical prompt suite freezeWhat identical prompt suite, with fixed option count strata and state length bands plus pinned revisions and item hashes, should freeze tokens per decision so that a new crossover measurement compares against published dollars per million without rerunning every baseline?
Calibrating sigmoid label scorers3 open
  • Per-bucket temperature coverage liftHow much selective-automation coverage does per-bucket temperature fitting buy on a GLiNER-style sigmoid checkpoint: fitting one temperature per option-count bucket and reporting threshold-realizable coverage at 5 percent error before and after, does the pilot's threefold coverage gap close or persist?
  • Single-label decision rule over sigmoidsWhich single-label decision rule over independent sigmoids gives the best risk-coverage tradeoff: argmax over raw sigmoids, argmax over renormalized sigmoids, or a temperature fitted directly on the argmax margin, and does the winner vary with option count?
  • Training change past the post-hoc ceilingWhen the post-hoc ceiling is reached, which training change improves selective coverage per unit cost on a frozen small encoder: focal or asymmetric loss substitution, proper-scoring-rule policy gradients, or few-shot example-augmented label representations without any weight updates?
Examples per label on fine-grained intents3 open
  • Frozen versus fine-tuned replicationAt matched examples per label on Banking77, how do frozen mean-prototype scoring, a frozen encoder with a trained head, and full encoder fine-tuning compare on micro-F1, macro-F1, and calibration, and does any frozen arm reproduce the GLiClass 8-shot recovery without gradient updates?
  • Selection versus count at fixed shotsAt fixed example counts, how much of few-shot intent accuracy comes from example selection and presentation (representative picks, retrieval routing, in-context formatting) versus count, and does smart routing beat naive scaling to 20 shots per label?
  • Calibration of few-shot headsDo few-shot adapted small decision encoders preserve usable selective-automation curves, or do 8-shot heads need per-shot-count temperature refits and abstention heads before their probabilities can gate automation?
Cardinality break point3 open
  • Measured per-trunk break pointsWhat are the measured per-trunk accuracy, ECE, and flip-rate break points on a fixed Banking77 cardinality sweep under pinned 512-token truncation with paired bootstrap intervals?
  • Relaxed truncation separation testDoes relaxing truncation to 2048 tokens and above move the 8K-class trunk breaks rightward while leaving DeBERTa-v3-large pinned, or do all four trunks move together under a shared head ceiling?
  • Label-length control token floorDoes a label-length control with short against paraphrased long wordings separate token-starvation breaks from marker-count breaks, and at what per-option token floor does accuracy recover per trunk?
Teacher calibration before distillation3 open
  • LoCa correction vs scaling on post-refit ECEDoes LoCa-style teacher correction improve student post-refit classwise-ECE at matched accuracy where pure teacher temperature scaling does not.
  • Minimal teacher-refit separation experimentWhat is the smallest teacher-student experiment that separates teacher-calibration effects from student-refit effects on post-refit ECE at high option counts.
  • ECE variant ranking sensitivityDo ECE variant choices change the ranking of teacher-calibration treatments for decision heads the way Nixon and coauthors report for recalibration methods.
Symmetric KL and tail dark knowledge3 open
  • Symmetric vs forward tail ranking after refitDoes symmetric KL preserve non-target tail ranking better than forward KL after the refit on a 77-way choice head at matched accuracy, measured by classwise-ECE and teacher-student tail correlation.
  • Reverse KL tail loss after refitDoes reverse KL lose non-target dark knowledge that the calibration refit cannot restore, or do forward, reverse, and symmetric students converge to the same post-refit calibration.
  • Skew interpolation per unit computeWhat skew interpolation between forward and reverse KL, if any, maximizes post-refit classwise calibration per unit of distillation compute on high-cardinality heads.
Velofy Decision Bench v03 open
  • Items per bucket for rank stabilityHow many items per bucket does v0 need for rank-stable comparisons, measured by rerunning the frozen suites with disjoint seeds and reporting rank agreement across model classes?
  • Deployment summary without poolingWhat single deployment summary of the v0 surface may a lab print (across budgets, suites, and load scenarios) without violating the no-pooling rule, so that two systems can still be ranked for one stated deployment?
  • Hidden-generator holdout designWhich hidden-generator holdout design best detects benchmark training, such as refusal-trigger memorization, without punishing legitimate fine-tuning on the public generators?
The smallest selection head that wins3 open
  • Minimal detached error predictorDoes a Kamath-style detached error predictor (top-k probabilities plus pooled features plus bucket identity, trained on held-out in-domain plus known-OOD items) beat per-bucket temperature-plus-threshold on risk-coverage for a small typed-decision encoder, and how small can its feature set get before the win disappears?
  • Abstain class and temperature groupingDoes a Deep Gamblers style m-plus-1 abstain class beat the post-hoc floor and the detached head at matched coverage on the same backbone, and does refitting temperatures on the answered-conditional distribution restore or change the optimal per-bucket grouping?
  • Cheapest novelty-rejection sliceHow much known-OOD or unanswerable training data does the winning head need to add novelty rejection without hurting ambiguity rejection, and does that transfer across held-out OOD families the way Kamath et al. report for QA?
Decision Bench v0, first run1 open
  • Fine-tuned models against the linear barAfter fine-tuning on the train split, where do Laya and a current GLiClass checkpoint land on accuracy and flip rate relative to the trained linear baseline?
Near-miss, per-bucket temperature, shortlists3 open
  • Fine-tuned Laya on near-miss optionsDoes a fine-tuned Laya checkpoint keep its accuracy under near-miss distractors, or does fine-tuning mostly teach easy separations?
  • A fine-tuned second stageWhen the second stage is fine-tuned on the same train split, does shortlist-then-decide beat the trained ranker alone?
  • An error-ranking signal for zero-shot modelsIs there a zero-shot confidence signal, such as top-two margin or cross-order agreement, that ranks errors at 20 options where temperature cannot?