Shortlist Then Rerank for High-Cardinality Typed Decisions

The question

This report asks what shortlist then rerank protocol preserves Banking77 accuracy and calibration past single pass limits while reporting shortlist recall separately. The parent paper in this graph, Option Cardinality Limits, establishes the failure: jointly encoded options degrade as label sets grow, Laya advises keeping choice questions under about 20 options, and Jev refuses above 255 options. Shortlisting is the obvious escape, and Laya already ships a predict_shortlist helper for it, but no fetched source in this graph defines the full protocol: which k, which shortlister, which reranker, how calibration survives the truncation, and which numbers must be reported so that a retrieval miss cannot hide inside a rerank accuracy.

Retrieval solved the analogous problem years ago with retrieve then rerank pipelines, and typed decisions have not yet imported its reporting discipline. This report does that import. It proposes a concrete protocol for Banking77 scale tasks, grounds each step in fetched primary sources, and states the measurement identity that makes the protocol honest: end to end accuracy equals shortlist recall times conditional rerank accuracy, so all three must be reported. It runs no new experiments, reports no new model results, and Velofy has not trained or shipped a decision model. Every Maestro Flash statement below is a proposal target.

First principles

End to end accuracy is a product of two probabilities. A two stage pipeline is right only if the gold label survives the shortlist and the reranker then picks it. So end to end accuracy equals recall at k times conditional accuracy. One accuracy figure cannot say which stage failed. The two factors can.

A joint pass spends one fixed budget across all options. When the state and every option share one sequence, each added option takes tokens and attention from the others. Cutting the option set from 77 to k gives each survivor more room. That is the first reason to shortlist. The second is that order sensitivity and miscalibration also grow with option count.

Truncation changes what a probability means. A softmax over k shortlisted labels is conditional on the shortlist. It says nothing about the labels that were cut, and a shortlist miss puts zero probability on the truth. Calibration therefore has to be fitted per k and reported twice: once for the reranker’s conditional distribution and once for the whole pipeline.

k is a trade, not a constant. A larger k raises recall but shrinks the per option budget and brings back position effects. A smaller k does the reverse. The right k sits where the marginal recall gain equals the marginal conditional loss, and only a sweep over k can find it.

Cost splits the same way. Label embeddings are computed once and cached, so the shortlist stage costs one query encoding plus 77 similarity scores. The rerank pass is the expensive stage, and its cost grows with k.

Background

Single pass typed decisions put the state and every option into one forward pass. The Laya teardown in this graph reads Laya as building one sequence from instructions plus masked option markers plus state, then scoring each marker with softmax. That design is fast and simple, and it breaks in two documented ways as labels grow. First, options share a fixed token budget: Laya README states 192 tokens on English and 256 on multilingual, which leaves about 3 to 4 tokens per label at 77 options. Second, order sensitivity and miscalibration grow with option count: Laya BENCHMARKS.md reports flip rates of 0.150 and 0.230 at 20 options on MASSIVE intent English, and as shipped expected calibration error (ECE) of 0.466 and 0.314. The Jev survey in this graph adds the closed model side: the independent decision-model-benchmark (DMB) reports Jev at 76.3 percent on 77 way banking with ECE 0.246, plus a hard refusal above 255 options. Past the single pass limit, the choice is to shard labels, decouple option encoding, or shortlist then rerank. This report covers only the third.

Retrieve then rerank is the standard answer in passage ranking, and its structure transfers directly. MS MARCO, which Bajaj et al. describe as 1,010,916 questions and 8,841,823 passages from 3,563,535 web documents, defines passage ranking as one of its three tasks. Nogueira and Cho describe a BERT passage reranker that took the top entry on the MS MARCO passage retrieval leaderboard, outperforming the previous best by 27 percent relative in mean reciprocal rank at 10 (MRR@10), and state of the art on TREC-CAR. The retriever side comes from dense dual encoders: Karpukhin et al. report that their Dense Passage Retrieval (DPR) model outperforms a strong Lucene BM25 system by 9 to 19 percent absolute in top 20 passage retrieval accuracy, and Reimers and Gurevych report cutting 10,000 sentence pairwise search from about 65 hours with BERT to about 5 seconds with Sentence-BERT (SBERT) while keeping accuracy. BEIR, which Thakur et al. describe as 18 datasets across diverse retrieval tasks, evaluates 10 systems and reports that reranking and late interaction models achieve the best zero shot performance on average, at high computational cost, while dense and sparse retrieval are more efficient but often underperform.

Two more findings shape the protocol. First, calibration needs its own treatment. Guo et al. report that modern neural networks are poorly calibrated and that temperature scaling, a single parameter variant of Platt scaling, is surprisingly effective at calibrating predictions. Laya BENCHMARKS.md confirms the recipe transfers to typed decisions, reporting that refitting one temperature per question type and option count takes ECE to 0.081 and 0.106. But a shortlist truncates the softmax denominator, so the reranker outputs a conditional distribution over k labels, and the temperature that calibrates a 77 way distribution need not calibrate a top 20 conditional one. Second, position effects should shrink when k shrinks. Liu et al. report that models use long contexts unevenly, with the worst performance when relevant information sits in the middle, and Zheng et al. report selection bias across 20 models and three benchmarks, attributing it to token bias toward specific option identifiers. A top k shortlist shortens the context and renumbers the identifiers, so the protocol measures flip rate at each k rather than assuming the improvement.

What the evidence says

The direct shortlist evidence for typed decisions is thin but concrete. Laya README documents predict_shortlist, which keeps the top k labels with a caller supplied embedding function and then runs one forward pass on that shortlist, with an example using the top 20 labels. Laya README states that embed_fn_from_agent mean pools the encoder already loaded on the agent, so the shortlister reuses the deployed backbone at no extra model cost, and Laya README states that probabilities on a shortlisted choice are over those k labels, with passthrough when k covers all labels. Laya README cites issue 102 as reporting that a top 20 zero shot shortlist moved a Banking77 run from 54.3 percent to 60.8 percent on the reporter setup, while noting the repository has not remeasured those figures. Even the unremeasured reporter gain suggests the shortlist recovers distinguishability that the fixed head budget destroys. But it is a single end to end number with no recall, no k sweep, and no calibration, which is exactly the reporting gap this protocol closes.

The vendor precedent for staging is TypeSafe itself. TypeSafe states that Jev supports cardinality up to 255, and TypeSafe adds that higher cardinality choices use a two stage system of independent scoring then explicit choice, which TypeSafe says causes occasional slowdown. The internal mechanism remains UNVERIFIED, and DMB independently reports the sharp exterior: 100 percent accuracy at 254 and 255 options on its code word suite, then 400 Too many choices at 256, 384, and 512, while every large language model (LLM) it tested handles 512. DMB reports Jev median (p50) latency of 264 to 276 ms flat from 2 to 255 options, which DMB describes as 1.2 times faster than gpt-oss-120b on Cerebras and 10 to 16 times faster than thinking mode models. A shortlist protocol for open models should therefore treat vendor two stage behavior as an existence proof, not a specification, and beat it on transparency: every stage measured, every miss attributed.

The third party Jev measurements set the baselines a protocol must decompose. AbdelStark reports Banking77 BTZSC at 72 labels with Jev at 0.870 versus GLiNER at 0.610, with a difference interval of [+0.220, +0.300], and AG News at 4 labels with Jev at 0.910 versus 0.700. AbdelStark notes that its pinned BTZSC configuration exposes 72 hypotheses and 200 rows with no positive candidate, which are excluded, so its denominator differs from the full 77 intents that Casanueva et al. describe over 13,083 examples. AbdelStark reports DAIR Emotion as the calibration warning: accuracy unresolved at 0.480 versus 0.440, but Brier 0.846 versus 0.668, negative log-likelihood (NLL) 5.588 versus 1.381, and zero probability on the true label for 16 percent of examples. AbdelStark notes that NLL is sensitive to exact zero probabilities and calls that intentional. That warning transfers directly to shortlists, because every shortlist miss assigns zero mass to the gold label by construction. On latency, AbdelStark reports GLiNER faster at small label counts (about 44 ms p50 versus 236 to 256 ms) and Jev slightly faster at 72 labels (246 ms versus 296 ms), while noting a local Apple M4 Max central processing unit (CPU) versus hosted Jev called from France. DMB adds the calibration and robustness baselines: ECE 0.246 for Jev against an LLM range of 0.039 to 0.122, 13 percent of choices changing under permutation against a worst LLM at 37 percent, and $0.07 per 1,000 decisions on banking against a cheapest LLM at $0.19.

The rerank training literature adds one design note. Zhuang et al. describe RankT5 with pairwise and listwise ranking losses, and report that listwise fine tuning gives better zero shot ranking on out of domain sets than classification losses. A typed decision reranker trained with listwise loss over shortlists may therefore generalize better to unseen label wordings than one trained with plain cross entropy, though no fetched source tests that claim on Banking77 style intent sets, so it stays a hypothesis. Tunstall et al. describe SetFit as contrastive siamese fine tuning of a sentence transformer followed by a classification head, with no prompts or verbalizers, and report comparable results to heavier few shot methods while training an order of magnitude faster. SetFit matters here as the shortlister upgrade path: a contrastively tuned bi-encoder over label text plus a few exemplars per intent, which Casanueva et al. show is viable with only 10 or 30 examples per intent on dual encoders trainable in minutes on a single CPU. The Hugging Face application programming interface (API) reports 251,354,399 downloads for sentence-transformers all-MiniLM-L6-v2 and 17,267,935 for BAAI bge-reranker-v2-m3, an adoption signal that the bi-encoder plus rerank stack is the deployed retrieval default. Hugging Face API lists PolyAI banking77 with 98 likes and 11,338 downloads.

Approach	Shortlist stage	Rerank stage	Reported Banking77 behavior	What is missing
Laya `predict_shortlist`	Caller embedding or mean pooled agent encoder, top 20 example	One joint forward pass over k, probabilities over k	Reporter figure 54.3 to 60.8 percent, not remeasured by repo	Recall at k, k sweep, conditional accuracy, calibration
Jev vendor two stage	UNVERIFIED, described as independent scoring	UNVERIFIED, described as explicit choice	DMB reports 76.3 percent at 77 way, flat latency to 255	Everything internal, stage boundary, recall
DPR plus monoBERT pattern	Dense dual encoder, 9 to 19 percent top 20 gain over BM25	BERT rerank, plus 27 percent relative MRR@10 on MS MARCO	Not run on Banking77 in any fetched source	Transfer to 77 intent classification, calibration
RankT5 listwise	Same retriever family	Listwise loss, better zero shot out of domain	Not run on Banking77 in any fetched source	Whether listwise beats cross entropy for intent rerank
SetFit tuned shortlister	Contrastive siamese sentence transformer, 10 to 30 shots viable per Casanueva et al.	Any joint head over k	Not combined with a joint rerank in any fetched source	End to end recall plus conditional curve

Analysis

The protocol has five stages, and each stage reports its own numbers so that misses are attributable. Stage zero freezes the evaluation. Use the full 77 intents that Casanueva et al. describe, not a 72 hypothesis subset, and pin the shortlister checkpoint revision, the label text templates, and the random seeds. AbdelStark pins its GLiNER revision and documents its 72 hypothesis exclusion, which is the right practice; the protocol additionally requires stating the denominator on every table, because 72 way and 77 way Banking77 figures do not compare. Fix a k grid before running, with 5, 10, 20, and 30 as defaults: 20 matches the Laya shortlist example and the soft threshold in Laya BENCHMARKS.md, while 5 and 30 bracket the recall knee from both sides.

Stage one shortlists and reports recall at k. Embed the query state and each of the 77 labels with a frozen bi-encoder, score by cosine similarity, and keep the top k. Label representations should be label text plus a short fixed template, with an optional few shot variant that averages in exemplar embeddings, following the SetFit and Casanueva et al. evidence that small per intent exemplar sets help dual encoders. The reported metric is recall at k: the fraction of items where the gold label survives the shortlist. This number is the ceiling on end to end accuracy, and it must appear in every results table, because a 60 percent end to end score means something entirely different at 65 percent recall (reranker nearly perfect) than at 95 percent recall (reranker losing a third of winnable items). Report recall at every k in the grid, plus recall at 77 as a sanity check that must equal 1.0.

Stage two reranks and reports conditional accuracy. Run one joint forward pass over the k shortlisted labels and take the argmax. Report conditional accuracy: accuracy computed only on items where the gold label is in the shortlist. The decomposition identity is then exact: end to end accuracy equals recall at k times conditional accuracy. Publish all three, never the product alone. This identity is what makes the protocol falsifiable. If recall at 20 is 0.90 and conditional accuracy is 0.85, end to end is 0.765, and the next point of gain must come from the shortlister, not the reranker. If recall is 0.98 and conditional accuracy is 0.62, the reranker is the bottleneck, and raising k will likely hurt through the token starvation and position effects that Laya README and Liu et al. document. Choose the operating k at the knee where marginal recall gain falls below marginal conditional loss, and show the full k sweep curve rather than a single picked k.

Stage three handles calibration explicitly, because shortlisting breaks naive calibration in two ways. First, the reranker distribution is conditional on the shortlist, so a temperature fit on full 77 way data does not transfer; refit one temperature per k on held out data, following Guo et al. and the Laya BENCHMARKS.md finding that per question type and option count refit is the highest value fix. Second, shortlist misses force zero mass on truth, which makes end to end NLL explode by the mechanism AbdelStark flags as intentional. The protocol therefore reports calibration twice. Report conditional ECE, Brier score, and NLL on the reranked distribution over items where gold survived, which measures the reranker honestly. Then report end to end Brier score and NLL with an explicit zero mass convention for misses, which measures the pipeline honestly. Never silently renormalize the shortlist distribution into a 77 way distribution by padding zeros without saying so; the padding is the measurement, not a detail. An optional third calibration object is a shortlist confidence interpolator that mixes retriever margin with reranker output, but no fetched source validates one for typed decisions, so it is exploratory until measured.

Stage four measures robustness and cost at each k. Permute shortlist order with fixed seeds and report the flip rate, defined as how often the argmax changes, alongside the reranked accuracy under permutation. Laya BENCHMARKS.md reports 0.150 and 0.230 at 20 options against a Jev measured 0.13, and DMB reports 13 percent for Jev against a worst LLM at 37 percent, so a shortlist protocol should show whether reranking 20 beats single passing 77 on this axis too, since Zheng et al. give reason to expect identifier bias to shrink with fewer options. Measure latency as three additive parts at each k: shortlist encode and score, rerank forward pass, and calibration overhead, reporting p50 and 95th percentile (p95). Cache label embeddings once, since SBERT style precomputation is the reason the shortlist stage can be near free at steady state, and state cache status on every latency table. Finally, price the pipeline per 1,000 decisions at the operating k for like for like comparison against the DMB figures of $0.07 for Jev and $0.19 cheapest LLM.

Implications for small decision models

For Jev, the protocol is a way to audit the vendor two stage claim from outside. TypeSafe states independent scoring then explicit choice with occasional slowdown, and DMB reports flat 264 to 276 ms latency with a sharp 400 refusal at 256. An external lab cannot see the stage boundary, but it can run the protocol against the Jev API as the reranker over client side shortlists at several k values, reporting recall, conditional accuracy, and conditional ECE at each k. If conditional accuracy at k 20 substantially exceeds the DMB 77 way figure of 76.3 percent, that bounds how much of Jev error is ranking versus full set handling. If conditional ECE stays near the DMB measured 0.246 across k, the miscalibration is in the scorer, not the set size. Either result is publishable, and neither requires vendor cooperation.

For Laya, the protocol turns an undocumented helper into a measured feature. Laya README documents predict_shortlist with a top 20 example and a reporter figure of 54.3 to 60.8 percent that the repository has not remeasured, while Laya BENCHMARKS.md advises keeping choice under about 20 options and reports temperature refit ECE of 0.081 and 0.106 with flip rates of 0.150 and 0.230 at 20 options. Running the protocol on Laya means remeasuring the reporter claim with recall at k, conditional accuracy, conditional versus end to end calibration, and flip rate at each k, on the full 77 labels. The likely outcome, given the 192 and 256 token head budgets that Laya README states, is that recall saturates early while conditional calibration needs per k temperatures, but that is a prediction to test, not a finding. Until it is measured, predict_shortlist is a plausible escape with one anecdote.

For Velofy’s proposed Maestro Flash, which has no weights and no results, the protocol is the acceptance test for the high cardinality story. The companion proposal targets separately encoded and cacheable option representations with per question type, option count, and language temperatures. The protocol validates or refutes that plan in order: first, does a frozen bi-encoder shortlist reach high recall at k 20 on full 77 way Banking77; second, does the joint rerank head beat the single pass baseline on conditional accuracy and flip rate at matched k; third, does per k temperature refit hold conditional ECE flat across the k grid; fourth, does the full pipeline beat the Laya 0.492 typed decisions figure and approach the third party Jev figures (AbdelStark 0.870 on 72 hypotheses, DMB 76.3 percent on 77 way) at stated latency and dollars per 1,000 decisions. If shortlist recall saturates below what end to end accuracy needs, the proposal must upgrade the shortlister (SetFit style tuning, exemplar augmented labels) before touching the reranker. The protocol makes that sequencing a measurement outcome rather than an opinion.

Limitations

No experiment was run for this report, so the protocol is unexecuted: the k knee, the recall ceiling, and the conditional calibration behavior on Banking77 are all UNVERIFIED until someone runs the grid. Every Jev number comes from vendor claims or two third party projects with different prompts, samples, and denominators (72 hypotheses in AbdelStark after exclusions versus 77 way in DMB), so cross benchmark comparison is approximate. Laya figures are self reported and were read from its README and BENCHMARKS.md, not reproduced here, and the 54.3 to 60.8 percent shortlist gain is a reporter figure the repository marks as not remeasured. The retrieval transfer assumes monoBERT, DPR, and RankT5 findings on passage ranking carry over to 77 way intent classification, which no fetched source tests. Calibration guidance assumes temperature refit per k suffices, but shortlist selection bias (the reranker only ever sees hard negatives that survived retrieval) may need a training change rather than a post hoc fix, which is also UNVERIFIED. Latency and cost comparisons are deployment specific, and download counts measure reuse rather than quality. Multilingual shortlist evidence is absent: no fetched source reports recall at k outside English for intent sets at this scale.

Open questions

At what k does frozen bi-encoder shortlist recall saturate on full 77 way Banking77, and does rerank conditional accuracy peak at the same k or at a smaller one.
How should calibration be reported and fitted when the shortlist truncates the distribution: is per k temperature refit on the conditional distribution sufficient, or does end to end selective automation need retriever margin interpolation or an explicit abstention rule on low shortlist margin.
Does reranking a shortlist reduce option order flip rate versus single pass full set scoring at matched end to end accuracy, and how many seeded permutations per item make that comparison stable across k.

References

← All papers