How Many Examples per Label Do Small Decision Encoders Need?

Status: REPORT. This paper synthesizes published papers, public READMEs, and public model and dataset application programming interfaces (APIs). It reports no new experiments. Velofy has not trained or shipped a decision model, and every Maestro Flash statement below is a proposal target. Every number below comes from a source fetched for this paper and is attributed inline.

The question

The parent paper in this graph, Zero-Shot Label Scoring in the GLiNER Family, ends with a question about examples per label: GLiClass reports that 8 examples per label recover its edge checkpoint on Banking77 from 0.0255 to 0.6644, and this paper asks whether that recovery pattern replicates on Banking77-scale tasks with frozen encoders and example-augmented label representations, or whether it requires fine-tuning. This report answers from published evidence only, and its short answer is that the recovery is real, the mechanism behind it is underspecified in the fetched paper text, and the surrounding literature already contains the experiment that would settle it.

Three facts frame everything below. First, the GLiClass paper text fetched for this report states the 8-shot result but no adaptation procedure for it: its evaluation section says only “We also report few-shot performance using 8 examples per label,” while its low-rank adaptation (LoRA) configurations belong to a logic and natural language inference (NLI) post-training stage, not to the few-shot evaluation. Whether Table 5 comes from gradient updates or from in-context examples is therefore UNVERIFIED from the paper text alone. Second, the GLiClass README documents an in-context few-shot mechanism (a special example token), which proves the frozen route is at least implemented, though not that it produced the reported table. Third, older work on the same dataset shows frozen encoders with a trained head already beating full BERT-Large fine-tuning at 10 shots per intent, which means the “requires fine-tuning” side of the question starts on the defensive. The rest of this report lays out the numbers, then designs the replication that closes the gap.

First principles

A label name alone carries little signal when labels are near neighbors. Banking77 separates intents such as reverted top-up and failed top-up. Their names differ by a single word, so a scorer that sees only the label text has very little to tell them apart. Examples supply the wording customers actually use, which is where the difference shows.

An example can change one of three things. It can train a head on top of fixed embeddings. It can move the encoder’s embedding space itself. Or it can sit in the input, or in an averaged label representation, with no weight changing at all. Each route has a different cost and a different failure mode. A few-shot result is not interpretable until it says which route produced it.

The frozen route works only if the space already separates the labels. Averaging a label’s examples into a prototype adds information but cannot reshape the space. If two intents overlap in a frozen encoder, their prototypes overlap too. A trained head can draw a sharper boundary in the same space. Fine-tuning can move the space.

Each extra example adds less than the one before. A prototype is the mean of N examples, and a mean settles quickly as N grows, so the first few examples carry most of the gain. That predicts a curve that is steep early and flat later. At small N, a single unrepresentative example also moves the mean a lot, so which examples are chosen matters as much as how many.

Examples are cheap relative to full data. Ten examples for each of 77 intents is 770 labeled rows (computed), against about 10k training examples for the full-data setting reported below. That gap is why examples per label is the right unit to measure.

Background

Banking77 is the fine-grained intent benchmark this question turns on. Casanueva et al. (arXiv:2003.04807, fetched abs page confirms the title “Efficient Intent Detection with Dual Sentence Encoders”) introduced it as a single-domain dataset of 13,083 customer service queries labeled with 77 intents, deliberately fine-grained (their example pair is reverted top-up versus failed top-up) to complement the coarser multi-domain sets: HWU64 with 25,716 examples for 64 intents in 21 domains, and CLINC150 with 150 intents and 23,700 examples across 10 domains. The Hugging Face dataset card for PolyAI/banking77 (fetched for this report) confirms the 13,083 query and 77 intent counts, describes the set as a very fine-grained single-domain intent collection, and records a CC-BY-4.0 license with 11,338 downloads and 98 likes. Larson et al. (arXiv:1909.02027, “An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction”) add the caveat that production intent systems also face queries outside every supported intent, a condition none of the few-shot numbers below measure.

Three adaptation families appear in the evidence, and keeping them separate is the main analytical work of this report. The first is frozen encoders with a trained head: sentence embeddings stay fixed and only a small classifier learns, as in the dual-encoder intent detectors. The second is encoder fine-tuning with a head: the encoder weights move on the few available examples before or while the head trains, as in SetFit. The third is in-context use of examples with no gradient updates at all: examples are placed in the input or used to build label representations at inference time, as in the GLiClass README mechanism, retrieval-augmented querying, and prototype-style matching.

The supporting cast is well documented in fetched abstracts. Reimers and Gurevych (arXiv:1908.10084, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks”) derive sentence embeddings from siamese and triplet structures that can be compared with cosine similarity, and report reducing the similar-pair search over 10,000 sentences from about 65 hours with BERT to about 5 seconds with Sentence-BERT. Snell et al. (arXiv:1703.05175, “Prototypical Networks for Few-shot Learning”) learn a metric space where classification is performed by computing distances to prototype representations of each class, which is the theoretical template for example-augmented label representations with no training at all. Schick and Schutze (arXiv:2001.07676, “Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference”) introduce Pattern-Exploiting Training, which reformulates inputs as cloze phrases, assigns soft labels to unlabeled examples, and then runs supervised training. Gao et al. (arXiv:2012.15723, “Making Pre-trained Language Models Better Few-shot Learners”) present LM-BFF for smaller models where fine-tuning is cheap, combining prompt-based fine-tuning with automated prompt generation. The pattern across these sources is that few-shot success comes from somewhere specific (the encoder, the head, the prompt, or the unlabeled data), and a report that does not say which one has not finished its job.

What the evidence says

The GLiClass 8-shot table is the finding to be explained. The GLiClass paper (arXiv:2508.07662, fetched abs page confirms the title “GLiClass: Generalist Lightweight Model for Sequence Classification Tasks”) reports, in the full text fetched for this report, few-shot performance using 8 examples per label across eight datasets (SST-5, Financial PhraseBank, AG News, Emotion, CAP SOTU, Rotten Tomatoes, MASSIVE, Banking77). The paper reports average gains over zero-shot of plus 0.1888 for edge (plus 50.0 percent), plus 0.2094 for modern-base (plus 47.1 percent), plus 0.1877 for modern-large (plus 36.1 percent), plus 0.1067 for base (plus 18.4 percent), and plus 0.1063 for large (plus 17.1 percent), and notes that smaller variants benefit disproportionately from limited supervision. Its discussion describes these as F1-score improvements of approximately 50 percent relative gain for the small variants, driven, in the paper words, by a joint text-label encoding strategy which leverages contextual interactions to generalize from minimal supervision.

The Banking77 column is the sharpest illustration of recovery from a collapsed zero-shot. The paper reports edge rising from 0.0255 at zero-shot to 0.6644 with 8 shots per label, modern-base from 0.3581 to 0.7683, modern-large from 0.3653 to 0.8354, and base from 0.4689 to 0.7975. The edge zero-shot figure sits near the floor for a 77-way task, so most of what the 8-shot number measures is rescue rather than refinement. The paper comparison table summarizes the capability as significant gains with minimal examples (plus 17 to 50 percent F1 with 8 examples per label), especially for smaller variants. The Hugging Face model API (fetched for this report) shows knowledgator/gliclass-base-v3.0 at 14,938 downloads under Apache-2.0.

The frozen-encoder evidence comes from the paper that created Banking77. Casanueva et al. keep two pretrained dual sentence encoders fixed, the Universal Sentence Encoder (USE) and ConveRT, and train only a multilayer perceptron (MLP) classifier (a single 512-dimensional hidden layer, stochastic gradient descent (SGD) with learning rate 0.7 and linear decay, dropout 0.75) on the resulting encodings, and they report that the detectors train in minutes on a single central processing unit (CPU). Their Table 3 (accuracy times 100, at 10 examples per intent, 30 per intent, and full data) reads, for Banking77: bert-fixed 67.55, 80.07, 87.19; bert-tuned 83.42, 90.03, 93.66; USE 84.23, 89.74, 92.81; ConveRT 83.32, 89.37, 93.01; USE plus ConveRT 85.19, 90.57, 93.36. Two facts matter. At 10 shots per intent, frozen USE plus ConveRT at 85.19 beats full BERT-Large fine-tuning at 83.42, so the frozen route wins at the data point closest to the GLiClass 8-shot setting. The same ordering holds on the other two sets (CLINC150 10-shot: 93.26 versus 91.93; HWU64 10-shot: 85.83 versus 84.86), so it is not a Banking77 accident. The paper notes the efficiency asymmetry explicitly: ConveRT uses 6 Transformer layers and 512-dimensional embeddings in a 59MB model against BERT-Large with 24 layers and 1024-dimensional embeddings, and the bert-tuned baseline trains for 50 epochs at the 10-shot setting, which is the fine-tuning cost the frozen route avoids.

The fine-tuning evidence comes from SetFit, with one large caveat. Tunstall et al. (arXiv:2209.11055, “Efficient Few-Shot Learning Without Prompts”) propose a two-step recipe: first fine-tune a pretrained Sentence Transformer on text pairs in a contrastive siamese manner, then train a classification head on the resulting embeddings. The encoder moves, so SetFit is evidence for the fine-tuning side of this report question, not the frozen side. The paper reports N equals 8 and N equals 64 labeled samples per class across six test datasets, and finds its 110M MPNet variant beats standard fine-tuning by an average of 19.3 points at N equals 8, with the gap shrinking to 5.6 points at N equals 64, and beats AdaPET by 4.0 and 1.5 points at the two settings. It reports training takes about 30 seconds at 0.025 dollars per split against about 700 seconds and 0.7 dollars per split for T-Few 3B, and lists model sizes of 355M, 110M, and 15M. The SetFit README (fetched for this report) adds the headline claim that with only 8 labeled examples per class on Customer Reviews, SetFit is competitive with fine-tuning RoBERTa Large on the full training set of 3k examples. The caveat is load-bearing: the fetched SetFit paper text contains zero mentions of Banking77, and its six evaluation sets have small label counts, so SetFit proves 8-shot fine-tuning works on coarse tasks and says nothing direct about 77-way intents.

The per-shot curve on Banking77 itself comes from Loukas et al. (arXiv:2311.06102, “Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking”), who frame contrastive methods as needing 10 to 20 examples per class and large language models (LLMs) as effective with 1 to 5. Their Table 4 (micro and macro F1 on the Banking77 test set, where micro-F1 in single-label classification reads as accuracy-like) gives the cleanest published answer to “how many”: SetFit on MPNet-v2 scores 57.4 and 55.9 at 1-shot, 76.7 and 75.9 at 3-shot, 83.5 and 83.3 at 5-shot, 88.1 and 88.1 at 10-shot, 90.6 and 90.5 at 15-shot, and 91.2 and 91.3 at 20-shot, against 94.1 and 94.1 for full-data fine-tuning on about 10k training examples. The curve is steep to 10 shots and flat after: going from 1 to 10 examples per label buys about 31 micro-F1 points, and the next 10 buy about 3. Selection matters as much as count: GPT-4 at 3-shot scores 83.1 and 82.7 with representative samples but 74.2 and 73.7 with random ones, and GPT-3.5 degrades going from 1-shot to 3-shot (75.2 and 74.3 down to 65.5 and 65.3 with representative samples), which suggests stuffing more examples into a fixed context can hurt on 77-label tasks. For baselines they report Mehri and Eric at 85.9 percent accuracy in the 10-shot setting and 93.8 percent full-data (secondhand via Loukas, UNVERIFIED against the original here), and Casanueva et al. at 93.6 percent full-data and 85.1 percent 10-shot. Their retrieval-augmented querying result is the closest published analogue to example-augmented label representations: retrieving only 2.2 percent of the examples used in classic few-shot settings while surpassing GPT-4 by 1.5 points and costing 700 dollars less on about 3k test examples.

Analysis

Four conclusions follow, stated carefully enough to act on.

First, the GLiClass 8-shot recovery is real and underspecified at the same time. The numbers are in the fetched paper text with exact gains and a full per-dataset table, so the effect is not in doubt as a reported result. But the fetched text contains no per-task adaptation procedure for that table: the evaluation section announces 8 examples per label without saying whether weights move, the LoRA table belongs to the logic and NLI post-training stage, and the only concrete few-shot mechanism in the fetched family sources is the README in-context interface with its special example token. A replication that assumes fine-tuning and one that assumes frozen in-context use are both consistent with the fetched evidence. Any citation of the 0.0255 to 0.6644 figure should carry the qualifier that the producing mechanism is UNVERIFIED.

Second, the frozen side already has a sufficient existence proof at 10 shots, but it trains a head, so the fully frozen question stays open. Casanueva et al. show fixed USE and ConveRT encodings with a trained MLP beating full BERT-Large fine-tuning on all three intent sets at 10 shots per intent, with training measured in minutes on one CPU. That settles “can frozen encoders compete at Banking77 scale” in the affirmative for the head-training variant. What it does not settle is the stricter variant this report asks about: frozen encoders with example-augmented label representations and no gradient updates at all (mean prototypes scored by cosine distance, or the GLiClass README in-context path). The nearest fetched numbers are secondhand (Mehri and Eric 85.9 at 10-shot via Loukas) or generative (GPT-4 3-shot 83.1 with representative picks), and neither is a small frozen encoder doing prototype matching on Banking77. That cell of the table is the experiment to run, not a result to cite.

Third, the count answer on Banking77 is approximately 8 to 10 examples per label for most of the gain, with selection quality as a second axis as large as count. Loukas et al. show SetFit crossing 88 micro-F1 at 10 shots against 94.1 full-data, with 5 shots already at 83.5 and 20 shots adding only about 3 points over 10. But the GPT-4 representative-versus-random gaps (about 6 to 9 micro-F1 points at matched shot counts) and the GPT-3.5 degradation from 1-shot to 3-shot show that which examples, and how they are presented, move results as much as adding examples does. Any replication that varies N without fixing a selection protocol will confound the two. The retrieval-augmented querying result sharpens the point from the cost side: retrieving 2.2 percent of examples beat the best dense-prompt method by 1.5 points at 700 dollars lower cost.

Fourth, pretraining-task match explains why frozen encoders work here and predicts where they stop working. USE and ConveRT are dual encoders pretrained on conversational response selection, which is near-neighbor to intent matching, and ConveRT does it in 6 layers and 59MB. The frozen route wins when the encoder already speaks the task dialect; BERT-Large fine-tuning has to learn that dialect from 10 examples per intent in 50 epochs and still loses. This predicts that frozen prototypes will work best on encoders with conversational or intent-adjacent pretraining and degrade on generic masked-language-model encoders, and that GLiClass-style joint text-label encoding (which the paper credits for generalizing from minimal supervision) is most valuable exactly where the frozen bi-encoder dialect is weakest: fine-grained label pairs like reverted versus failed top-up that need cross-attention to separate. The sibling cardinality analysis in Option Cardinality reaches the same fork from the token-budget side: joint encoding buys interaction at small label counts and loses to decoupled label representations as labels grow.

Implications for small decision models

For Jev, the implications are behavioral only, since its architecture and training are undisclosed. The sibling survey Jev and the Typed-Decision Landscape reports third-party Banking77 measurements in the 76 to 87 percent band with a 255-choice cap, which sits below the fetched frozen-encoder and SetFit 10-shot figures (85.19 and 88.1) but above the GLiClass edge 8-shot figure (0.6644). The honest reading is that Jev zero-shot behavior is being compared against few-shot numbers: Jev per-label example support is UNVERIFIED, so no fetched evidence says whether its Banking77 band would move under an 8-shot protocol.

For Laya, the evidence converts into a diagnostic ordering. The sibling teardown Laya: An Open Typed-Decision Engine, Examined reports Banking77 at 0.425 for both checkpoints and attributes it to the option-token budget (about 3 to 4 tokens per label at 77 options), with base checkpoints near chance before fine-tuning. Before accepting an architectural verdict, Laya-style systems should add per-label few-shot examples and re-measure, since GLiClass edge gained about 0.64 F1 on Banking77 from 8 shots without any architectural change. If examples close little of the gap, the head budget is the binding constraint and the fix is decoupled label representations. If examples close most of it, the encoder was fine and the label text was the weak signal.

For Velofy’s proposed Maestro Flash, which has no weights and no results, the evidence becomes a measurement protocol the proposal must commit to. Run Banking77 at N in {0, 1, 3, 5, 8, 10, 20} per intent with fixed test data and fixed seeds, in three arms: frozen encoder with mean-prototype cosine scoring (zero training), frozen encoder with a trained head (the Casanueva arm), and encoder fine-tuning with a head (the SetFit arm). Report micro and macro F1 plus a calibration curve at each N, since no fetched few-shot source reports expected calibration error (ECE), Brier score, or negative log-likelihood (NLL), and selective-automation behavior under few-shot heads is UNVERIFIED everywhere in this report. Add a representative-selection arm at fixed N, since Loukas et al. show selection moving results by 6 to 9 points. The pre-registered decision rule: if the zero-training prototype arm reaches within a few points of fine-tuning at N equals 8, ship example-augmented label representations as the default path; if it trails while the trained-head arm matches fine-tuning, ship frozen encoders with per-deployment head fits. None of these statements is a Maestro Flash result. Each is a test the proposal must pass, alongside the backbone evidence in Encoder Backbones 2026 and the interface analysis in Maestro Flash.

Limitations

This report inherits every limitation of its sources. All accuracy and F1 numbers are self-reported by their papers, read from full texts fetched for this report but not reproduced here. Cross-row comparison in the table is approximate: Casanueva et al. report accuracy while Loukas et al. report micro and macro F1 (accuracy-like in single-label classification but not identical in every implementation), and the GLiClass table metric is F1 per the paper discussion, so small gaps between rows are not decisive. The GLiClass 8-shot adaptation procedure is UNVERIFIED from the fetched text, which is a gap in the primary source, not just in this report. The Mehri and Eric figures are secondhand via Loukas et al. and were not checked against the original. No fetched source reports calibration, confidence quality, or selective-automation curves for any few-shot intent adapter, so the calibration column of this question is empty. No fetched source runs prototype-only matching with a small frozen encoder on Banking77, so the central replication cell has no number. CLINC150 out-of-scope behavior under few-shot adaptation is unmeasured in every fetched few-shot source. No experiment was run for this report.

Open questions

1. At matched examples per label on Banking77, how do frozen mean-prototype scoring, a frozen encoder with a trained head, and full encoder fine-tuning compare on micro-F1, macro-F1, and calibration, and does any frozen arm reproduce the GLiClass 8-shot recovery without gradient updates?
2. At fixed example counts, how much of few-shot intent accuracy comes from example selection and presentation (representative picks, retrieval routing, in-context formatting) versus count, and does smart routing beat naive scaling to 20 shots per label?
3. Do few-shot adapted small decision encoders preserve usable selective-automation curves, or do 8-shot heads need per-shot-count temperature refits and abstention heads before their probabilities can gate automation?

References

1. Stepanov et al., GLiClass: Generalist Lightweight Model for Sequence Classification Tasks, https://arxiv.org/abs/2508.07662
2. GLiClass repository README (Knowledgator/GLiClass), https://github.com/Knowledgator/GLiClass
3. Casanueva et al., Efficient Intent Detection with Dual Sentence Encoders, https://arxiv.org/abs/2003.04807
4. PolyAI/banking77 dataset card, https://huggingface.co/datasets/PolyAI/banking77
5. Tunstall et al., SetFit: Efficient Few-Shot Learning Without Prompts, https://arxiv.org/abs/2209.11055
6. SetFit repository README (huggingface/setfit), https://github.com/huggingface/setfit
7. Loukas et al., Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking, https://arxiv.org/abs/2311.06102
8. Reimers and Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, https://arxiv.org/abs/1908.10084
9. Snell et al., Prototypical Networks for Few-shot Learning, https://arxiv.org/abs/1703.05175
10. Larson et al., An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction, https://arxiv.org/abs/1909.02027
11. Schick and Schutze, Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference, https://arxiv.org/abs/2001.07676
12. Gao et al., Making Pre-trained Language Models Better Few-shot Learners, https://arxiv.org/abs/2012.15723

← All papers