What Multilinguality Costs on Indic Decisions, and How to Measure It

The question

What is the measured cost of multilinguality on native-text Indic decision benchmarks for mmBERT-base against an English ModernBERT control behind a router, scored per language rather than averaged? This report was commissioned as the answer to that question, which arrives as an open question from the backbone survey.

The short answer is that no direct measurement was located. No fetched source evaluates mmBERT-base against an English ModernBERT checkpoint behind a router on native-text Indic decision tasks with per-language reporting, and this report ran no new evaluations. What follows is therefore a bound, not a reading: the adjacent evidence that constrains the answer, the confounders that would corrupt a naive measurement, and a concrete protocol for producing the number. A sibling proposal in this graph specifies the dataset that would make the native-text half of that protocol possible (see the Indic typed-decision benchmark proposal), so this report does not repeat its construction plan and concentrates on the comparison itself. Nothing here claims Velofy has trained or shipped any model. Every number below is attributed inline to the fetched source that reports it.

First principles

The two trunks share a body and differ in vocabulary. The fetched configs give mmBERT-base and ModernBERT-base the same width (768), depth (22 layers), and attention layout. What differs is the training data and the embedding table: 256,000 tokens against 50,368. At width 768 that table is about 197M parameters for mmBERT-base and about 39M for ModernBERT-base (computed). What remains is about 110M parameters in both models (reported by the mmBERT model card, computed for ModernBERT-base). So mmBERT-base serves more than 1,800 languages through a trunk the same size as the one ModernBERT-base spends on English.

A fixed trunk spread over more languages gives each one less. Adding languages can help through shared structure, which is the positive transfer XLM-R reports. It can also hurt, because the same non-embedding parameters must now encode many scripts and vocabularies. Which effect wins is an empirical question, and it can differ by language.

The cost is per language, so averages hide it. A user in one language gets that language’s accuracy, not the mean. A surplus in Hindi does not fix an error in Marathi. Any honest measurement therefore reports each language on its own row.

A router decides where the cost is paid. If the router keeps English on the English trunk, English traffic pays no multilingual cost, and the cost falls only on the requests sent to mmBERT-base. A misrouted request adds a cost of its own, and confidence cannot catch it afterward: the English checkpoint in Laya’s README scores 0.000 on Khmer at 0.952 confidence. Measuring the trunk without the router’s own errors measures the wrong system.

A benchmark measures the trunk only if the text is native. Translated test sets carry their own artifacts, and models react to them. A gap measured on translated text mixes trunk quality with translation quality.

Background

The cost of multilinguality is the performance a multilingual model gives up, per language, relative to a dedicated model, in exchange for covering many languages in one checkpoint. The XLM-R paper frames it as a tradeoff between positive transfer (related languages helping each other) and capacity dilution (a fixed parameter budget spread thinner), and its abstract reports both sides: training on 100 languages with more than 2 terabytes of filtered CommonCrawl data, gains of 14.6 percent average accuracy on XNLI, 13 percent average F1 on MLQA, and 2.4 percent F1 on named entity recognition (NER) over multilingual BERT, alongside per-language figures such as 15.7 percent XNLI improvement on Swahili and 11.4 percent on Urdu. Its abstract closes with the striking claim that multilingual modeling need not sacrifice per-language performance, reporting XLM-R as very competitive with strong monolingual models on GLUE and XNLI. A recent survey of multilingual large language models names the curse of multilinguality as one of their significant limitations across encoder-only, decoder-only, and encoder-decoder families. So the literature contains both the worry and a qualified reassurance, and the qualifier matters: XLM-R’s reassurance was measured at 100 languages, while mmBERT claims more than 1,800.

The Indic-specific version of the worry is stated most crisply in the MuRIL paper abstract. MuRIL’s authors argue that multilingual models trained on more than 100 languages together give Indic languages a small share of vocabulary and training data, cite prior work for degraded effectiveness in resource-lean scenarios, and add two phenomena that flat-multilingual training handles poorly: Latin-script transliteration of Indic text and code-mixing with English. Their response was a model trained on Indic text corpora only, augmented with translated and transliterated document pairs, which their abstract reports as significantly outperforming multilingual BERT on all tasks in the XTREME benchmark. Its diagnosis is the hypothesis this report tests: shared capacity across thousands of languages should cost something on any single Indic language, and averaging across languages is exactly how that cost hides.

The two trunks in the question make the comparison unusually clean, because they share an architecture. Fetched config files show mmBERT-base and ModernBERT-base as identical in every trunk hyperparameter: hidden size 768, 22 layers, 12 attention heads, 8,192 maximum position embeddings, local attention over 128 tokens, global attention every 3 layers, and intermediate size 1152. They differ in what they were trained on and in vocabulary: ModernBERT’s release blog reports English sources (web documents, code, and scientific articles), 2 trillion tokens with mostly unique data, and 149M parameters at base and 395M at large, while mmBERT’s abstract reports 3 trillion tokens of multilingual text in over 1,800 languages with more than 1,700 low-resource languages added only during a decay phase. The fetched configs report vocabulary sizes of 256,000 for mmBERT-base against 50,368 for ModernBERT-base, and mmBERT’s model card reports 307M total parameters with only 110M non-embedding parameters, a Gemma 2 tokenizer, and MIT licensing. (Laya’s README describes the same checkpoint as 322M, presumably counting decision heads differently; compare parameter counts on one harness, not across accountings.) The table below collects the comparison. Download counts come from the Hugging Face application programming interface (API).

The router half of the question is not hypothetical machinery. Laya ships a Router that picks between its English ModernBERT-large checkpoint (421M parameters, 512 truncation) and its multilingual mmBERT-base checkpoint (322M, 1024 truncation) per request, and its README documents why routing must happen before the forward pass: the English checkpoint scores 0.000 accuracy at 0.952 confidence on Khmer, so confidence gating cannot recover from a misrouted request. The README reports script detection in under 0.5 ms of pure Python, and notes a 20,000-text English check in which no English prose was moved off the English checkpoint. The control arm is therefore a real, shippable pattern: route by script and language heuristics, keep English on the ModernBERT control, and send the rest to mmBERT-base.

The scoring half of the question, per language rather than averaged, also has precedent. The XTREME paper abstract warns of a wide spread of results across languages even as English numbers reach human performance, and its README requires per-language prediction files for leaderboard submission. XLM-R’s abstract reports per-language gains rather than only averages. In the other direction, the IndicXTREME paper abstract reports its headline as a model averaging 2 points of absolute improvement across languages and tasks, which is precisely the kind of average under which a per-language cost can hide. Laya’s README shows the same hazard in decision-shaped form: across 51 languages its English checkpoint macro-averages 0.227 accuracy with macro expected calibration error (ECE) 0.733, only 23 of 51 languages clear 3 times random, and Khmer sits at 0.000 with 95.2 percent confidence. Any average over that table understates the worst deployed experience.

What the evidence says

There are two candidate evidence pools for the question: multilingual benchmarks with Indic coverage, and Indic-specific benchmarks. Both exist, but neither contains the head-to-head this question asks for, and their text provenance varies in ways that matter for the native-text qualifier.

The multilingual pool starts with XNLI, whose abstract reports 15 languages including Hindi and Urdu, built by extending the MultiNLI development and test sets, with translate-test reported as the best available baseline. XTREME covers 40 languages and 9 tasks with zero-shot transfer from English as its evaluation setting, and its README lists Bengali, Hindi, Marathi, Tamil, Telugu, and Urdu among its language codes. XTREME-R extends this to 10 tasks and 50 languages with a diagnostic suite. MASSIVE is the most decision-shaped of the set: its abstract reports 1M realistic parallel virtual-assistant utterances across 51 languages, 18 domains, 60 intents, and 55 slots, with modeling results on XLM-R and mT5 covering exact match accuracy, intent classification accuracy, and slot-filling F1. But MASSIVE was created by professional translators localizing the English-only SLURP dataset into 50 languages, so it is localized text, not native text. FLORES-101 is explicit about the same property: its abstract reports 3,001 sentences extracted from English Wikipedia and translated into 101 languages by professional translators.

The Indic-specific pool is led by IndicXTREME, whose paper abstract reports 9 tasks across 20 languages with 105 evaluation sets, 52 of them new, backed by the IndicCorp corpus of 20.9B tokens over 24 languages from 4 families. Its repository README breaks the 9 tasks into 5 sentence classification, 2 structure prediction, 1 question answering (QA), and 1 retrieval task, and names the components: IndicCOPA (the COPA test set manually translated into 18 Indic languages), IndicQA (manually curated cloze-style reading comprehension in 11 languages), IndicXParaphrase (10 languages), IndicSentiment (13 languages), IndicXNLI (automatically translated XNLI in 11 languages), Naamapadam (NER with manually curated test sets for 9 languages), MASSIVE (7 Indic languages), and FLORES (18 Indic languages). The README also reports the accompanying IndicBERT v2 model at 278M parameters supporting 23 Indic languages plus English. IndicGenBench targets generation rather than decisions: its abstract reports 29 Indic languages across 13 scripts and 4 families, with the largest PaLM-2 models best on most tasks but a significant gap against English in all languages. It evidences Indic coverage gaps at every scale, but it cannot score a typed-decision head.

Provenance is the axis that decides which of these suites can carry the native-text qualifier, and the suites sort into three strata:

Why provenance changes the answer is established by the translation-artifacts literature. The Artetxe et al. paper on translation artifacts in cross-lingual transfer reports that translating premise and hypothesis independently reduces their lexical overlap, to which current models are highly sensitive, and that correcting for such artifacts improves XNLI state of the art by 4.3 points for translate-test and 2.8 points for zero-shot approaches. Points-scale swings from translation handling alone mean a translated benchmark cannot cleanly separate trunk quality from transfer luck. The XTREME README’s own warning, that its automatic pseudo test sets are noisy and should not be treated as ground truth, shows the benchmark authors agreeing.

Against this background, mmBERT’s self-reported claims are strong but one-sided. Its abstract reports dramatic gains from decay-phase-only low-resource data and classification performance similar to OpenAI’s o3 and Google’s Gemini 2.5 Pro despite the short exposure, plus significant outperformance of previous-generation models on classification and retrieval in high- and low-resource languages. Its model card repeats the outperformance claim against XLM-R on classification, embedding, and retrieval, and adds that mmBERT is significantly faster than any previous multilingual encoder. None of these claims is a comparison against an English ModernBERT control on Indic decisions, none is scored per Indic language in any fetched source, and none has been independently confirmed in this survey. The honest ledger entry is therefore: vendor-reported multilingual strength, UNVERIFIED as a multilinguality-cost measurement.

The direct cell in the evidence matrix, mmBERT-base versus ModernBERT plus router on native-text Indic decision tasks, is empty in every source fetched for this report. That emptiness is itself the finding the rest of this report works with.

Analysis

Four bounds constrain the unmeasured cost, one confounder threatens any measurement of it, and one piece of arithmetic decides where the cost is actually paid.

First, the XLM-R monolingual-parity claim bounds the English-side cost near zero, but weakly. XLM-R’s abstract reports competitiveness with strong monolingual models on GLUE and XNLI at 100 languages. If that transferred to mmBERT at 1,800 plus languages, routing English to the ModernBERT control would be nearly free insurance. But language count grew roughly 18 fold between the two claims while trunk capacity stayed in the same band (mmBERT-base reports 110M non-embedding parameters), and capacity dilution is explicitly one half of XLM-R’s own tradeoff framing. A parity result at 100 languages does not license a parity assumption at 1,800. Treat the English-side cost as small but unmeasured, not as zero.

Second, MuRIL bounds the Indic-side cost from above in the opposite direction. Its abstract reports outperforming multilingual BERT on all XTREME tasks with a model trained on Indic corpora only, which is what winning back the dilution looks like at fixed size: concentrate the vocabulary and the data on the languages being served. The comparison is mBERT-era, so it does not transfer numerically to ModernBERT-class trunks, but its mechanism does. With identical 768-wide, 22-layer trunks and a 256,000-token vocabulary against a 50,368-token one, mmBERT-base must represent at least 1,800 languages through the same depth that ModernBERT devotes to English, and its embeddings account for roughly two thirds of its 307M reported parameters. The dilution is not a metaphor here; it is visible in the parameter accounting.

Third, Laya’s README bounds the value of the router, which caps the damage any trunk-level cost can do in deployment. The Khmer figure (0.000 accuracy at 0.952 confidence) and the 51-language macro table (0.227 accuracy, 0.733 macro ECE) describe what happens without routing: confident wrong answers on unseen scripts. A router that detects script in under 0.5 ms and demonstrably leaves 20,000 English texts unmoved converts a trunk-quality question into a traffic-split question. The multilinguality cost is then paid only on the routed fraction, while English keeps control-trunk quality. This is why the question’s phrasing, control behind a router rather than control alone, is the right experimental design: it matches the deployable system.

Fourth, the translation-artifacts result bounds the measurement error of the translated stratum at points scale. A 4.3-point translate-test swing and a 2.8-point zero-shot swing on XNLI from handling translation better means any head-to-head scored on IndicXNLI or other machine-translated suites could flip its verdict on provenance handling rather than trunk quality. The protocol implication is to stratify: report native or manually curated suites (IndicQA, Naamapadam) as the headline, professionally localized suites (MASSIVE intent classification, IndicCOPA) as the decision-shaped secondary, and machine-translated suites (IndicXNLI) as a contrast arm that quantifies the translationese effect rather than the trunk effect.

The confounder is averaging. XTREME’s abstract reports a wide spread across languages, XLM-R’s abstract leads its low-resource case with per-language figures, and Laya’s per-language table shows a macro average coexisting with a zero. IndicBERT v2’s headline, 2 points averaged across languages and tasks, cannot tell a deployer whether Kannada or Malayalam improved at all. For typed decisions the per-language requirement is stronger than a reporting preference: a support ticket misrouted in Marathi is a user who does not get help, and no Hindi surplus compensates them. Score accuracy, Brier score, ECE, median (p50) latency, and flip rate separately for each language, and treat any average as a summary of a table the reader can still inspect.

The arithmetic that locates the cost is the traffic split. Let the router send fraction f of traffic to mmBERT-base and keep 1 minus f on the ModernBERT control. The system-level multilinguality cost is f times the per-language trunk gap on routed languages, plus the misrouting cost (control-trunk errors on traffic that should have been routed, and vice versa). Laya’s README documents both failure directions receiving attention: non-English Latin text that previously fell through to English now reaches the multilingual checkpoint, and script ranges the router does not know no longer default to English. A measurement that omits the router’s confusion matrix therefore misattributes routing errors to trunk quality. Log the router decision with every scored example.

A minimal protocol has three arms (mmBERT-base alone, ModernBERT control plus router, and IndicBERT v2 at 278M as an Indic-dedicated reference, per its README), four metric families per language (discrimination, calibration, latency, and flip rate), and three provenance strata reported separately. Fine-tune each arm identically on English task data for the zero-shot suites, following XTREME’s documented transfer setting, and additionally report any suite permitting native-language fine-tuning. Until that protocol runs, the responsible quantitative statement is a bound: English-side cost plausibly small by XLM-R analogy but unmeasured at 1,800-language scale, Indic-side cost positive by MuRIL mechanism with magnitude unknown, and translated-suite readings uncertain at points scale by the artifacts literature.

Implications for small decision models like Jev, Laya, and Velofy’s proposed Maestro Flash

For Jev, the implication is behavioral and limited. Jev is a closed API with undisclosed architecture, as documented in the Jev survey, so no trunk-level cost can be attributed to it from public evidence. Its independently reported figures (Banking77 accuracy, ECE, flip rate) remain legitimate comparison targets, but they are single-system readings, not multilinguality-cost measurements.

For Laya, this report supports the shipped architecture while sharpening its evaluation gap. The ModernBERT-large plus mmBERT-base pair behind a script router is exactly the design this report’s question presupposes, and the README’s per-language table across 51 languages is already the reporting granularity this report asks for. The gaps are provenance and calibration depth: the Indic rows sit on localized or translated suites rather than native text (the sibling benchmark proposal details the pattern from Laya’s own tables), and the Laya teardown reports mean as-shipped ECE of 0.314 for multilingual with no fitted temperatures shipped. The highest-leverage Laya-adjacent measurement is therefore narrow: fit multilingual temperatures, then re-score the Indic rows per language on the native stratum.

For Velofy’s proposed Maestro Flash, the implication is a stage-zero gating requirement. The proposal’s mmBERT-base production candidate survives this report, but the ablation set must include the English ModernBERT control behind the router from day one, scored per language on the native stratum. Concretely: add router-confusion-matrix logging, the three provenance strata, and per-language calibration metrics to the harness the backbone survey specifies, and make the trunk decision conditional on the routed-system reading rather than on any averaged leaderboard. If mmBERT-base wins only on breadth while losing calibration or latency on the core languages, the fallback is proven in Laya’s shipped code: route by language instead of forcing one trunk. No training has been done for Maestro Flash, and nothing here predicts which arm wins.

Limitations

This is a literature and artifact report, not a benchmark. No model was trained, fine-tuned, or measured for it, no Velofy checkpoint exists, and its central quantitative cell is explicitly empty. Every head-to-head figure it discusses comes from author or vendor sources: mmBERT’s classification and retrieval claims from its own abstract and model card, ModernBERT’s pretraining figures from its own paper and release blog, MuRIL’s XTREME outperformance from its own abstract, Laya’s routing and accuracy figures from its own README. Independent reproduction on a shared harness with per-language reporting is the missing step throughout.

Download counts (400,800 for mmBERT-base, 4,372,504 for ModernBERT-base, per Hugging Face API queries run for this report) measure adoption and packaging maturity, not quality. License tags (MIT for mmBERT-base per its card and API metadata, Apache-2.0 for ModernBERT-base per the API) should be rechecked before commercial use. Architectural identity between the two base trunks is established only at the config-file level; training recipes, data mixtures, and tokenizer behavior (notably the Gemma 2 tokenizer on the mmBERT side, per its card) differ, so equal depth does not mean equal capacity per language.

Several quantities that matter most are UNVERIFIED at the level this question needs: per-language Indic decision accuracy for either trunk, per-language calibration of either pretrained checkpoint, router confusion rates on real Indic traffic including transliterated and code-mixed input, and latency of either arm on the reader’s own hardware. The sibling proposal covers transliteration and code-mixing as construction requirements; this report adds only that MuRIL’s abstract flags both as known failure modes of flat-multilingual models, which makes their absence from current suites a gap rather than a detail. Finally, benchmark coverage of Indic languages remains thin even before provenance filtering: XNLI covers Hindi and Urdu among 15, and XTREME’s 40-language list names only a handful of Indic codes, so per-language conclusions will rest on few languages until native suites grow.

Open questions

1. What is the per-language multilinguality cost of mmBERT-base against an English ModernBERT control behind a router on the native stratum (IndicQA, Naamapadam, and decision-shaped native suites as they appear), scored separately for accuracy, Brier score, ECE, p50 latency, and option-order flip rate?
2. How much of any measured gap is translationese rather than trunk quality, estimated by scoring the same decision tasks in native, professionally localized, and machine-translated forms and reporting the provenance spread alongside the trunk spread?
3. At what router misrouting rate does the routed system’s total cost (trunk gap on routed traffic plus misrouting errors) exceed the single-trunk alternative, and does that crossover differ between high-resource Indic languages and low-resource ones?

References

1. mmBERT paper, arXiv:2509.06888
2. ModernBERT paper, arXiv:2412.13663
3. XLM-R paper, arXiv:1911.02116
4. XTREME benchmark paper, arXiv:2003.11080
5. XTREME-R paper, arXiv:2104.07412
6. XNLI paper, arXiv:1809.05053
7. MuRIL paper, arXiv:2103.10730
8. IndicXTREME and IndicBERT v2 paper, arXiv:2212.05409
9. MASSIVE dataset paper, arXiv:2204.08582
10. Translation artifacts in cross-lingual transfer paper, arXiv:2004.04721
11. FLORES-101 benchmark paper, arXiv:2106.03193
12. IndicGenBench paper, arXiv:2404.16816
13. Multilingual LLMs and curse of multilinguality survey, arXiv:2406.10602
14. mmBERT-base model card, https://huggingface.co/jhu-clsp/mmBERT-base
15. ModernBERT release blog, https://huggingface.co/blog/modernbert
16. Laya repository, https://github.com/NandhaKishorM/laya
17. IndicBERT repository (IndicXTREME benchmark details), https://github.com/AI4Bharat/indicbert
18. XTREME benchmark repository, https://github.com/google-research/xtreme

← All papers