Zero-Shot Label Scoring in the GLiNER Family

Status: SURVEY. This paper synthesizes published papers, public READMEs, and public model application programming interfaces (APIs) for the GLiNER family of models. It reports no new experiments. Velofy has not trained or shipped a decision model, and every Maestro Flash statement below is a proposal target. Every number below comes from a source fetched for this paper and is attributed inline.

The question

This paper asks what the GLiNER family of models teaches about zero-shot label scoring: the problem of assigning a score to each of an arbitrary, caller-supplied set of labels in one encoder pass, without generating text. The parent survey in this graph defines typed decisions as closed, schema-typed questions answered without autoregressive generation, and the companion paper on option cardinality studies what happens as label sets grow. This paper does not repeat those reports. It focuses on the one family of open models that has worked on exactly this problem the longest, and asks which of its design choices transfer to small decision models such as Jev, Laya, and Velofy’s proposed Maestro Flash.

The GLiNER family matters here for three reasons. First, it predates the current typed-decision framing by several years, so its results are independent evidence rather than a response to it. Second, it is fully open (papers, code, and checkpoints), so its mechanisms are inspectable in a way closed systems are not. Third, it has already confronted, in public, several problems this graph treats as open: label-count scaling, multilingual label scoring, and the choice between joint and separate encoding of text and labels.

First principles

A zero-shot label scorer has three parts: an encoder that turns text into vectors, a way to turn each label into a vector, and a function that compares the two and returns a score. Every design choice in this family is a choice about one of those parts, under five constraints.

• A fixed output head cannot take new labels. A classifier with one output per class learns its class set at training time. To accept a label set the caller picks at inference time, the label must become an input, text the encoder reads, rather than a row in a weight matrix. GLiNER takes that step, and the rest of the family inherits it.
• Reading labels and text together gets more expensive as labels grow. In a joint (uni-encoder) design, labels and text share one input sequence. Every added label lengthens the sequence, takes context window away from the text, and raises the cost of self-attention, which grows with the square of sequence length. Encoding labels separately, with label vectors computed once and reused, removes both costs but gives up direct label-text interaction inside the encoder. That is the tradeoff this paper tracks.
• A matcher learns to say no only if it sees labels that do not apply. With a per-label yes-or-no loss, training only on labels present in the text pushes the model to treat every label it sees as present. Negative labels are part of the recipe, not an extra.
• Per-label sigmoid scores are not a distribution. Each label gets its own probability between 0 and 1, and the scores need not sum to one. That suits multi-label output, but nothing in the objective makes scores comparable across a label set, and a fixed 0.5 threshold is a convention, not a guarantee.
• In zero-shot use, the label string is all the model knows about a class. When classes differ in fine detail, a few examples per label carry information a short label string cannot.

The evidence and the four lessons below bear on these constraints in turn: label-count scaling, joint versus decoupled encoding, negative sampling, calibration, and few-shot recovery.

Background

GLiNER was introduced by Zaratiana et al. (arXiv:2311.08526, November 2023) as a compact named entity recognition (NER) model that identifies arbitrary entity types. The paper’s core move is to reframe open NER as matching entity type embeddings to textual span representations in latent space rather than as a generation task. The input is a single sequence combining entity types expressed in natural language with the text to extract from. The paper states that each entity type is preceded by a learned [ENT] token and that a [SEP] token separates the label block from the text. A bidirectional encoder (BERT or DeBERTa class) produces token representations, entity embeddings pass through a feedforward network, span representations are computed for candidate spans, and a matching score between the two is computed with a dot product followed by sigmoid activation. Training uses binary cross-entropy (BCE), so the paper notes the score can be read as the probability that a given span has a given type.

Training data came from Pile-NER, a dataset the paper attributes to Zhou et al. and describes as derived from the Pile corpus. The paper reports three sizes: GLiNER-S at 50M parameters, GLiNER-M at 90M, and GLiNER-L at 0.3B. Two training details are worth naming because they recur across the family. The paper reports an ablation on negative entity type sampling in which 0 percent negatives yields 53.3 F1, 50 percent yields 60.9, and 75 percent yields 58.6 on its 20-dataset suite, which establishes that training with labels absent from the text is load-bearing for zero-shot behavior. The paper also reports that even BERT-base reaches around 49 F1 on its out-of-domain benchmark, about 2 F1 points above the average it reports for ChatGPT and InstructUIE, which is an early data point that a small bidirectional encoder with the right training recipe can beat a much larger generative model at scoring arbitrary labels.

GLiClass (Stepanov et al., arXiv:2508.07662, August 2025) adapts the GLiNER architecture from span extraction to sequence classification. The paper’s diagnosis of the status quo is explicit: generative large language models (LLMs) suffer from inconsistent instruction following and computational inefficiency, cross-encoders must process text-label pairs sequentially (which the paper calls the bottleneck for large label sets), and embedding-based approaches struggle with logical and semantic constraints. GLiClass keeps GLiNER’s central idea, namely that labels are first-class inputs expressed in natural language, and applies it to whole-text classification with single-label and multi-label heads. The primary architecture is a uni-encoder in which text and labels are processed jointly in one encoder, but the paper states it also explored bi-encoder, fused bi-encoder, and encoder-decoder variants. Backbones are DeBERTa v3, and the paper reports that DeBERTa-based models consistently outperformed ModernBERT-based ones in its experiments. The paper adds three mechanisms this graph has not otherwise encountered in open decision models: layer-wise attention re-weighting via a squeeze-excitation scheme across encoder layers, a temperature parameter inside the scorer, and an adaptation of proximal policy optimization (PPO) for multi-label classification aimed at data-sparse conditions or human feedback.

GLiNER2 (Zaratiana et al., arXiv:2507.18546, July 2025) unifies NER, text classification, and hierarchical structured extraction in one model behind a schema-driven interface, while keeping central processing unit (CPU) efficiency and compact size as explicit goals. The GLiNER2 README (fastino-ai/gliner2, fetched for this paper) documents two architectures: the span architecture of GLiNER2, and the boundary architecture of GLiNER2.5, which replaces a fixed span-width grid with sparse start and end pairing so that spans of any length fitting the encoded window are representable. The same README lists six checkpoints with sizes from 74M (2.5-small, DeBERTa-v3-xsmall) to 340M (2-large, DeBERTa-v3-large), plus personally identifiable information (PII) and guardrail fine-tunes, and recommends 2.5-base (194M) for English and 2.5-multi (287M, mDeBERTa-v3-base) for multilingual use.

A further member extends the picture: the GLiNER bi-encoder paper (Stepanov et al., arXiv:2602.18487, February 2026) decouples label encoding from context encoding to remove the context-window bottleneck of joint encoding.

What the evidence says

The zero-shot evidence starts with the original GLiNER paper. On its out-of-domain NER suite, the paper reports that the 90M medium model matches UniNER-13B at 55 F1 for both despite being about 140 times smaller, and that the 0.3B large model outperforms UniNER-13B by an average of 5 points. The paper’s abstract claims GLiNER outperforms ChatGPT and fine-tuned LLMs in zero-shot evaluation on various NER benchmarks. These are NER-span numbers, not classification numbers, so they transfer to typed decisions only as evidence about the scoring mechanism, not as a claim about any decision benchmark. The paper also documents a weakness: on non-Latin languages it reports particularly poor results, with Bengali at 0.89 F1, which foreshadows the multilingual calibration problems this graph’s parent papers document for later systems.

GLiClass reports the most directly relevant classification numbers. Its Table 2 (read via the paper full text) lists five v3.0 checkpoints with average F1 and throughput in examples per second (ex/s): edge at 32.7M parameters and 0.4900 average at 97.29 ex/s, modern-base at 151M and 0.5577 at 54.46 ex/s, modern-large at 399M and 0.6197 at 43.80 ex/s, base at 187M and 0.6764 at 51.61 ex/s, and large at 439M and 0.7193 at 25.22 ex/s. Three facts stand out. Accuracy scales with size within each backbone family. DeBERTa v3 beats ModernBERT at matched scale (base 0.6764 versus modern-large 0.6197 despite fewer than half the parameters), which the paper notes explicitly. And the speed range is about 4x from fastest to slowest, which makes the accuracy-throughput tradeoff a genuine frontier rather than a single dominant point. The GLiClass README adds the headline efficiency claim that classification runs about 10 times faster than cross-encoder models by using a single forward pass.

The same paper reports few-shot behavior that matters for decision-model deployment. With 8 examples per label, the paper reports average gains of plus 0.1888 for edge (plus 50.0 percent), plus 0.2094 for modern-base (plus 47.1 percent), and plus 0.1877 for modern-large (plus 36.1 percent). The Banking77 column is the sharpest illustration: the paper reports edge rising from 0.0255 zero-shot to 0.6644 with 8 shots per label. That near-zero zero-shot figure on a 77-label intent task, inside a family whose headline claim is zero-shot generality, deserves emphasis. It suggests that zero-shot label scoring degrades badly on large, fine-grained label sets even here, and that a handful of examples per label recovers most of the loss. The GLiClass README documents the corresponding affordances: in-context examples via a special token, per-task prompts, hierarchical labels in dot notation, and a chunking pipeline with configurable text and label chunk sizes for long documents.

The scaling evidence comes from the bi-encoder paper. Its abstract states that joint encoding suffers quadratic complexity as entity labels increase, and reports two headline figures: 61.5 percent micro-F1 zero-shot on the CrossNER benchmark, and up to 130 times throughput improvement at 1024 labels against uni-encoder predecessors by using pre-computed label embeddings. The GLiNER README corroborates the qualitative claim, stating that the bi-encoder variant pre-computes label embeddings and scales to more than 100 entity types without degradation. GLiNER-BioMed adds that synthetic LLM-distilled pre-training plus general-domain fine-tuning yields a 5.96 percent F1 improvement over its strongest baseline at p less than 0.001, which is evidence that the zero-shot recipe benefits from domain-targeted synthetic data rather than from scale alone.

Adoption evidence comes from the Hugging Face model API (fetched for this paper). The GLiNER2 family dominates: the API reports 445,822 downloads for fastino/gliner2-base-v1, 243,120 downloads and 240 likes for fastino/gliner2.5-multi-v1, and 202,278 for fastino/gliner2-multi-v1, against 14,938 downloads for the most downloaded GLiClass checkpoint (knowledgator/gliclass-base-v3.0). Download counts measure reuse, not quality, but the gap between the extraction checkpoints and the classification checkpoints is a real signal about where the ecosystem’s demand has concentrated so far.

The independent benchmark evidence connects the family to this graph’s reference systems. The AbdelStark pilot (jev-benchmarks README, fetched for this paper) compares hosted Jev against fastino/gliner2.5-multi-v1 on 300 held-out examples with 100 per condition. It reports Jev at 0.910 versus GLiNER2.5 at 0.700 on 4-label AG News, Jev at 0.870 versus 0.610 on 72-label Banking77/BTZSC, and an unresolved accuracy tie on 6-label DAIR Emotion (0.480 versus 0.440) where Jev is substantially worse calibrated: Brier score 0.846 versus 0.668, negative log-likelihood (NLL) 5.588 versus 1.381, and zero probability on the true label for 16 percent of examples. On latency it reports GLiNER2.5 faster on small label sets (about 44 ms median (p50) on a local CPU versus 236 to 256 ms hosted) but Jev slightly faster at 72 labels (246 ms versus 296 ms p50). On selective automation it reports Jev covering 0.830 and 0.860 of examples at 5 percent error on the first two conditions against 0.240 and 0.270 for GLiNER2.5. No fetched GLiNER-family source reports calibration for its own checkpoints, whether as expected calibration error (ECE), Brier score, or NLL, so the calibration comparison runs in one direction only: Jev’s miscalibration is measured, GLiNER2.5’s calibration is UNVERIFIED beyond this pilot’s Brier and NLL figures.

Analysis

Four lessons follow, each specific enough to act on.

First, natural-language labels scored by a learned matcher are a proven substitute for fixed output heads. Every member of the family takes labels as text at inference time ([ENT]-prefixed entity types in GLiNER, label strings in GLiClass, schema fields in GLiNER2) and scores them against the input with a small learned head over shared encoder representations. No fetched source in the family trains one classifier head per task. The mechanism differs by output shape (span matching for extraction, pooled text-to-label scoring for classification), but the interface is the same: the caller supplies the label set, the model scores it. That is exactly the typed-decision interface, arrived at independently from the NER side. The negative-sampling ablation (53.3 to 60.9 F1 from adding absent labels in training) shows the training recipe must expose the model to labels that do not match, or the matcher never learns to say no.

Second, joint encoding buys accuracy at small label counts and loses to decoupled encoding as labels grow, and the family has now published both sides of that tradeoff. GLiClass chose the uni-encoder as its primary design for rich text-label interaction, and its numbers show the cost: throughput falls from 97.29 to 25.22 ex/s across the size range, and the 72-label condition in the independent pilot is the one place the hosted API beats the local encoder on latency (246 vs 296 ms). The bi-encoder paper then quantifies the alternative: pre-computed label embeddings with up to 130x throughput at 1024 labels and no context-window competition between labels. The companion cardinality paper in this graph reaches the same conclusion from Laya’s token-budget arithmetic. The GLiNER family adds the missing half of that argument: a published, measured demonstration that decoupling preserves zero-shot quality (61.5 percent micro-F1 on CrossNER) while removing the bottleneck. The open design question is where the crossover sits for classification-shaped decisions rather than NER spans, since no fetched source reports a full accuracy-versus-label-count curve for GLiClass.

Third, few-shot examples are doing more work than the zero-shot framing admits. GLiClass’s 8-shot gains (plus 36 to 50 percent relative) and especially the Banking77 edge figure (0.0255 to 0.6644) show that on fine-grained intent labels, the zero-shot matcher is near chance and a few examples per label fix it. This rhymes with Laya’s documented near-chance zero-shot on its own benchmark (0.362 and 0.342 against a 0.461 majority baseline, per this graph’s parent survey) and its reliance on fine-tuning for its headline number. The pattern across both families is that inference-time label text alone is a weak signal for fine-grained distinctions, and that a small number of labeled examples per label is the cheapest known remedy. For decision-model evaluation this means zero-shot and few-shot numbers must always be reported as a pair; a zero-shot-only comparison confounds the scoring mechanism with the label-text quality.

Fourth, calibration is the family’s documented gap. GLiNER-family scores are per-label sigmoid outputs trained with binary cross-entropy, and every usage example in the fetched READMEs applies a fixed 0.5 threshold. No fetched family source reports ECE, Brier score, NLL, or any temperature-fitting procedure for its own checkpoints (GLiClass has a temperature parameter in the scorer, but no fetched source reports fitted values or calibration curves). The independent pilot therefore supplies the only calibration numbers available: GLiNER2.5’s Brier 0.668 and NLL 1.381 on DAIR Emotion beat Jev’s 0.846 and 5.588, but its coverage at 5 percent error (0.240 and 0.270) trails Jev badly (0.830 and 0.860), which means its probabilities rank worse for selective automation despite better average scores. A family that already tunes thresholds by hand and ships no calibration recipe is the natural audience for this graph’s proper-scoring-rule training work, not a competitor to it.

Implications for small decision models

For Jev, the pilot comparison is the whole story available: Jev’s accuracy lead over GLiNER2.5 grows with label count (plus 0.210 at 4 labels, plus 0.260 at 72 labels, tie at 6 labels on a hard emotion task), while its calibration lead does not hold (worse Brier and NLL on DAIR Emotion, and zero probability on truth for 16 percent of examples). Since Jev’s architecture is undisclosed and no training paper exists, the mechanism behind its large-label-count advantage is UNVERIFIED; the GLiNER evidence suggests candidates (decoupled label encoding, few-shot-style label descriptions, better negative sampling) but cannot confirm any of them. The practical implication is narrow: teams choosing between a hosted decision API and a local GLiNER checkpoint should benchmark on their own label count, because the two systems’ latency ordering flips between small and large sets in the pilot’s measurements.

For Laya, the GLiNER family is a prior-art check on three documented weaknesses. On high-cardinality options, Laya’s joint marker encoding with a fixed head budget (192 or 256 tokens, about 3 to 4 tokens per label at 77 options, per this graph’s parent survey) is the same joint-encoding bottleneck the bi-encoder paper measures and removes; pre-computed, cacheable label representations are the family’s validated answer. On zero-shot quality, Laya’s near-chance base numbers and GLiClass’s 0.0255 Banking77 edge number are the same phenomenon in two families, and GLiClass’s 8-shot recovery to 0.6644 suggests Laya-style systems should benchmark few-shot label examples before concluding the encoder is at fault. On multilingual coverage, GLiNER’s Bengali 0.89 F1 and Laya’s Khmer 0.000 accuracy at 0.952 confidence (per the parent survey) are the same warning from opposite ends: multilingual backbones do not automatically yield calibrated multilingual decisions, and routing or per-language calibration remains necessary.

For Velofy’s proposed Maestro Flash, which has no weights and no results, the family supplies design constraints rather than conclusions. Separately encoded and cacheable option representations are now supported by two independent lines of evidence (this family’s bi-encoder measurements and the cardinality paper’s token-budget analysis). Per-label few-shot examples should be a first-class input, given that 8 shots per label recovered GLiClass edge from 0.0255 to 0.6644 on Banking77. Calibration must be built in from the start, because the family shows that good zero-shot ranking (GLiNER2.5’s pilot Brier beating Jev’s) does not imply usable selective-automation curves (its coverage trailing threefold). And DeBERTa v3 beating ModernBERT inside GLiClass’s controlled comparison is a caution against assuming the newest backbone is the best scorer; backbone choice for Maestro Flash should be settled by a matched scoring experiment, not by inheritance. None of these statements is a Maestro Flash result. Each is a test the proposal must pass.

Limitations

This paper inherits every limitation of its sources. All GLiNER-family accuracy numbers are self-reported by the family’s own papers, read from full texts fetched for this paper but not reproduced here. The pilot comparison rests on 300 examples with 100 per condition, which supports the paper’s reported bootstrap intervals but not per-class claims at 72 labels. The two Banking77-scale figures in this graph do not share a denominator (72 hypotheses after exclusions in the pilot versus 77 intents in other suites), so cross-paper comparison at large label counts is approximate. Latency figures mix local CPU, local GPU throughput, and hosted API measurements and are operational observations, not normalized timings. No fetched source reports a calibration curve, an accuracy-versus-label-count curve, or an option-order sensitivity measurement for any GLiNER-family checkpoint, so three of this graph’s central questions have no family-side answer. Jev internals remain UNVERIFIED throughout. No experiment was run for this survey.

Open questions

1. Where is the joint-versus-decoupled crossover for classification-shaped decisions: at what label count, label-text length, and backbone size does a bi-encoder with pre-computed label embeddings match or beat a uni-encoder on accuracy per millisecond, and does the GLiNER-family NER crossover transfer?
2. What is the cheapest calibration recipe that gives a GLiNER-style per-label sigmoid scorer usable selective-automation curves: does per-option-count temperature fitting suffice, or does the independent-versus-softmax structure of the scores require a proper-scoring-rule training change?
3. How many examples per label does a small decision encoder need on fine-grained intent sets: can the GLiClass 8-shot recovery pattern be replicated on Banking77-scale tasks with frozen encoders and example-augmented label representations, or does it require fine-tuning?

References

1. Zaratiana et al., GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer, https://arxiv.org/abs/2311.08526
2. Stepanov et al., GLiClass: Generalist Lightweight Model for Sequence Classification Tasks, https://arxiv.org/abs/2508.07662
3. Zaratiana et al., GLiNER2: An Efficient Multi-Task Information Extraction System with Schema-Driven Interface, https://arxiv.org/abs/2507.18546
4. Stepanov et al., The Million-Label NER: Breaking Scale Barriers with GLiNER bi-encoder, https://arxiv.org/abs/2602.18487
5. Yazdani et al., GLiNER-BioMed: A Suite of Efficient Models for Open Biomedical Named Entity Recognition, https://arxiv.org/abs/2504.00676
6. GLiNER repository README (urchade/GLiNER), https://github.com/urchade/GLiNER
7. GLiClass repository README (Knowledgator/GLiClass), https://github.com/Knowledgator/GLiClass
8. GLiNER2 repository README (fastino-ai/gliner2), https://github.com/fastino-ai/gliner2
9. GLiNER official documentation, https://urchade.github.io/GLiNER/
10. Hugging Face model API, GLiNER family search and fastino/gliner2.5-multi-v1 model card, https://huggingface.co/models?search=gliner
11. AbdelStark jev-benchmarks pilot comparing Jev with GLiNER2.5, https://github.com/AbdelStark/jev-benchmarks
12. nibzard decision-model-benchmark (DMB), https://github.com/nibzard/decision-model-benchmark

← All papers