SearchBook a call

RESEARCH / PAPERS

proposal

An Indic-First Typed-Decision Benchmark: Dataset Proposal

Proposal. This describes planned work; any numbers marked TARGET are goals, not measured results.

A proposal for a human-verified benchmark of typed decisions in Indic languages, with native-script and transliterated pairs for each core example.

Velofy Lab ·

Status: DATASET PROPOSAL. No data collection described in this paper has been carried out. This paper specifies what would be built and why, with motivation cited from existing published benchmarks and from the documented Indic-language weaknesses of a currently shipping open system, laya. No number in this paper is a measurement of the proposed dataset, because the dataset does not yet exist.

velofy.co research report, r3. 2026-09-23.

Abstract

Existing multilingual evaluation of typed decisions treats Indic languages as a handful of rows in a much larger table. MASSIVE (arXiv:2204.08582) covers Hindi, Bengali, Tamil, Telugu, Malayalam, Kannada, and Urdu among 51 languages in total. Cross-lingual natural language inference (XNLI, arXiv:1809.05053) covers Hindi and Urdu among 15. Neither was designed to test the typed-decision primitives (choice, score, noul, span, json) that systems like laya (github.com/NandhaKishorM/laya) and the proposed Maestro Flash target. Neither includes the transliterated, code-mixed input that is extremely common in real Indic-language product traffic. IndicXTREME (arXiv:2212.05409) is a stronger Indic-specific effort, but it targets classification, natural language inference (NLI) and question answering (QA) in general, not typed decisions. IndicGenBench (arXiv:2404.16816) targets open-ended generation, the opposite end of the spectrum.

We propose a dedicated benchmark: human-verified (not solely machine-translated) typed-decision examples across at least 6 major scheduled Indic languages, plus a Latin-script transliterated variant of each, spanning all of Maestro Flash’s proposed output primitives and built from the same application-workflow themes laya already uses for its English-centric benchmark. The motivation comes straight from laya’s own documented Indic-language numbers: every Indic language laya benchmarks sits well below its English number, even on its multilingual checkpoint. The construction method combines translation with IndicTrans2, transliteration with a pipeline in the style of MuRIL (Multilingual Representations for Indian Languages), human verification, and seed examples written by native speakers. It is designed to avoid the “translationese” pitfall that the authors of MASSIVE and IndicXTREME explicitly guarded against in their own construction.

First principles

A benchmark score is an average over items. It can only measure what its items contain, and it hides whatever its average pools together. Five consequences shape this proposal.

  • Competence is held per language, per script and per task. A model that routes English support tickets well may not route Marathi ones. A model that reads Hindi in Devanagari may not read the same Hindi typed in Latin letters. A benchmark speaks only for the cells it actually covers.
  • Translated text is not native text. A translation carries traces of its source language’s word order and phrasing. A model scored on translated items is partly scored on how well it reads translations, not on how well it reads what users write. Native authoring, or native review of every translated item, is how the evaluation split stays honest.
  • A pooled average hides the weak group. If one language family is strong and another weak, the pooled number can still look acceptable. Reporting per language and per family keeps the weak cell visible.
  • A label is only as reliable as the agreement behind it. If two native speakers disagree about an item’s answer, no model can be scored cleanly on it. Inter-annotator agreement bounds how precisely accuracy can be measured, which is why this benchmark has two native speakers label each verified item and reports their agreement (§4.4).
  • Item counts set the error bar. A small per-language set gives a wide interval, and a gap between two languages means something only when both sets are large enough to separate it from noise. That is why annotation budget limits the first tranche to six core languages (§4.1, §6).

1. Introduction

India has 22 scheduled languages. Per the 2011 census (the most recent full census with detailed language data, cited as general background rather than as a paper reference), several hundred million people speak languages that get essentially no dedicated typed-decision evaluation in the current literature. This matters for the product category this project targets, typed decisions over arbitrary input state. Typed-decision systems are disproportionately deployed in domains such as customer support, financial services, government services and healthcare triage, where serving users in their language of comfort, not just a language they can technically read, has direct material consequences. A support ticket misrouted because a choice head could not reliably classify Marathi input is not merely a lower accuracy number. It is a customer who does not get help.

Existing evaluation that comes close to typed decisions does not close this gap. Laya’s own BENCHMARKS.md, read directly for this review, gives the clearest evidence of the gap in a currently shipping system. On MASSIVE intent classification (20 options, random baseline 0.050), laya’s multilingual checkpoint scores 0.430 on Hindi against 0.680 on English (macro across both of laya’s checkpoints, from laya’s per-language table). It scores 0.220 on Telugu, 0.240 on Malayalam and 0.150 on Kannada. Every Indic language in laya’s per-language table sits below its English number on both checkpoints, and several sit close to the noise floor even at 20-way classification. These are laya’s own measured, published, self-reported numbers, read directly from its BENCHMARKS.md, not a claim this paper makes about laya.

The literature offers partial tools for measuring this gap, but no complete one. MASSIVE (arXiv:2204.08582) is professionally localized, not machine-translated, into its 51 languages, which is methodologically strong. But its label taxonomy (60 intents, 55 slots) is fixed and does not map onto the typed-decision primitives this project uses: MASSIVE has no ordinal score task, no span extraction task and no json schema task at all. XNLI (arXiv:1809.05053) is also professionally translated, and is limited to one three-way classification task (entailment, neutral, contradiction). IndicXTREME (arXiv:2212.05409) is the strongest existing Indic-specific effort: 9 tasks across 18 to 20 languages, explicitly designed to test zero-shot transfer, with most of its 105 evaluation sets newly contributed rather than translated from an English benchmark. But its task suite (classification, NLI and QA-style tasks) does not include the typed-decision primitives this project needs. It also lacks transliterated and code-mixed variants, which are common in real Indic-language digital communication (WhatsApp messages, support chat, social media) but rare in formally translated benchmark text. IndicGenBench (arXiv:2404.16816) evaluates the opposite capability, open-ended generation quality, and does not apply to closed typed-decision evaluation at all.

This paper proposes a benchmark that closes this specific gap: typed decisions, native-verified Indic text, transliterated variants included, covering the full primitive set. We call it the Indic Typed-Decision Benchmark (ITDB), proposed here as a dataset construction plan, not yet executed.

Multilingual benchmarks close to typed decisions. MASSIVE (arXiv:2204.08582) and XNLI (arXiv:1809.05053) are the two benchmarks laya itself reports against, and the two most directly comparable existing resources; §1 details their limits for this project. Banking77 (arXiv:2003.04807) is English-only. It is cited as the methodological precedent for fine-grained intent taxonomies drawn from a real domain (77 banking intents), which ITDB’s choice construction (§4) follows for its Indic-language financial-services and support-triage sub-tasks.

Indic-specific benchmarks and models. IndicXTREME (arXiv:2212.05409) and its companion IndicCorp corpus (20.9B tokens, 24 languages) are the closest existing effort, and the main template ITDB follows for construction rigor: native-speaker annotation, and avoiding pure machine translation where possible. IndicGenBench (arXiv:2404.16816) is cited as a contrasting example of where Indic natural language processing (NLP) evaluation has concentrated (generation), against where this paper argues it is under-served (typed decisions). MuRIL (arXiv:2103.10730) supplies the transliteration-augmentation method that ITDB’s transliterated variants (§4.3) follow directly. MuRIL’s authors explicitly built translated and transliterated document pairs as a training signal; this paper adapts the same pairing idea to building evaluation data. IndicTrans2 (arXiv:2305.16307) is the proposed machine translation (MT) tool for translating the seed data (§4.3). It was chosen because it covers all 22 scheduled Indian languages with an open, permissively licensed model, and reports strong results against prior Indic MT systems in its own paper. Paramanu (arXiv:2401.18034) is cited as evidence that morphologically rich Indic languages can benefit from treatment specific to a language family rather than flat multilingual treatment. That supports ITDB’s decision to stratify by language family in construction and reporting (§4, §5), not only by individual language.

The system this benchmark is built to evaluate. Laya (github.com/NandhaKishorM/laya) is the concrete, currently shipping system whose documented Indic-language numbers (§1) motivate this proposal. It is the baseline against which any future Maestro Flash evaluation on ITDB would be compared, alongside laya itself if it can be run on the released dataset.

3. Design principles

Four principles, each responding to a specific limitation identified in §1 and §2:

  1. Native to typed decisions, not repurposed classification. Every ITDB example is written directly as a choice, score, noul, span, or json question, in the exact schema that Maestro Flash and laya’s own application programming interfaces (APIs) use (question type, instructions, criteria). It is not adapted after the fact from a classification dataset’s label column. This ensures the benchmark tests the primitives in question, including span and json (JavaScript Object Notation) primitives that no existing Indic benchmark surveyed here covers.
  2. Native verification, not translation alone. MASSIVE’s stated method is professional localization rather than raw machine translation, and IndicXTREME emphasizes newly contributed rather than translated evaluation sets. Following both, ITDB’s evaluation split must be either written by native speakers or professionally reviewed and corrected after machine translation, with the translation tool recorded for each example. The training-augmentation split may legitimately use machine translation at larger scale, per the multilingual and Indic data pipeline in the Maestro Flash technical report.
  3. Transliteration as a first-class input variant, not an afterthought. Following MuRIL’s explicit transliteration-augmentation design (arXiv:2103.10730), every native-script example in ITDB’s core languages should have a paired Latin-script transliterated variant. Transliterated and code-mixed text is extremely common in real product traffic (support chat, social media) and essentially absent from MASSIVE, XNLI and IndicXTREME.
  4. Language-family stratification in reporting. Paramanu (arXiv:2401.18034) gives evidence that morphologically rich languages may need different treatment than a flat multilingual average implies. ITDB’s reporting (§5) therefore breaks results out by language family (Indo-Aryan: Hindi, Bengali, Marathi, Gujarati, Urdu, Punjabi, Odia; Dravidian: Tamil, Telugu, Kannada, Malayalam) as well as giving a single pooled number, so a strong pooled score cannot mask a systematic Dravidian-language weakness or the reverse. Laya’s own per-language table (§1) already hints at this failure mode. On laya’s multilingual checkpoint, its Dravidian-language numbers (Telugu 0.220, Malayalam 0.240, Kannada 0.150, Tamil 0.230) are, as a cluster, weaker than its Indo-Aryan numbers (Hindi 0.430, Bengali 0.290). This is an observation from a small per-language table in one system’s benchmark, not a validated general claim about language families. ITDB is proposed precisely to test whether the pattern holds under dedicated, larger, native-verified evaluation.

4. Proposed construction methodology

4.1 Language and task coverage

Core languages (native script, human-verified): Hindi, Bengali, Marathi, Tamil, Telugu, Kannada. These six cover both major language families (Indo-Aryan and Dravidian) with three languages each. They are a first tranche rather than all 22 scheduled languages at once, given realistic annotation budgets (§6).

Extended languages (machine-translated and lightly reviewed, a lower verification bar, clearly labeled as such in the released data): Gujarati, Urdu, Punjabi, Odia, Malayalam. These five extend coverage while being explicit that their verification level is below the core six, following the practice of clearly separating translated data from data written or verified by humans.

Transliterated variants: Latin-script transliterated pairs for all core-language examples, built following MuRIL’s method.

Task and primitive coverage mirrors the application-workflow themes laya’s own BENCHMARKS.md already defines and benchmarks in English (support triage, email and phishing detection, moderation, guardrail and jailbreak detection, retrieval-augmented generation (RAG) passage relevance, model routing). Indic-language results are then directly comparable to laya’s published English numbers in task construction, if not in raw data:

  • choice: department or intent routing (following laya’s support-triage theme, 10-way), with a higher-cardinality variant following Banking77’s fine-grained-intent precedent (arXiv:2003.04807).
  • score: urgency or severity rating on a fixed rubric (following laya’s security-incidents theme).
  • noul: phishing or spam detection, churn-risk detection, refund-request detection (following laya’s email and phishing and customer-service themes directly).
  • span: extraction of key entities (dates, amounts, reference numbers) from support tickets and invoices in each covered language. This is the one primitive with no Indic-language typed-decision precedent identified in this review.
  • json: structured extraction of a small, flat invoice or ticket schema (following laya’s invoice-processing theme). Again, no existing Indic precedent was identified.

4.2 Seed generation

English seed examples are drawn from the same public sources laya’s own training and evaluation mix uses, where licenses are compatible (per the description of its training mix in laya’s README and BENCHMARKS; the exact source datasets are to be checked against their individual licenses at execution time). Newly written examples are added for the span and json primitives, which have no direct precedent in an existing dataset.

4.3 Translation and transliteration

Seed examples are machine-translated into all eleven target languages with IndicTrans2 (arXiv:2305.16307). For the six core languages, native speakers then review and correct the machine translations before they enter the evaluation split. This human-in-the-loop pass, not raw machine output, is the human-verification step that principle 2 (§3) requires. Latin-script variants of each core-language example are produced with a transliteration model or a rule-based scheme, chosen at execution time. The starting candidates are MuRIL’s own transliteration pipeline, if released, or a comparable open tool such as the AI4Bharat transliteration toolkit, following the pairing method MuRIL’s authors describe (arXiv:2103.10730).

4.4 Quality control

Following IndicXTREME’s stated emphasis on rigor in benchmark construction (arXiv:2212.05409), each human-verified example should have at least two independent native-speaker annotations for label agreement, with disagreements settled by a third annotator. Inter-annotator agreement (for example, Cohen’s kappa for choice and noul, weighted kappa for ordinal score) should be reported per language and per task in the released dataset documentation. No number in this paper reports it yet, because no annotation has been carried out.

5. Proposed evaluation protocol

Metrics mirror the rest of this project’s evaluation plan (see the experimental plan in the Maestro Flash technical report): accuracy; soft accuracy where a teacher distribution is available; Brier score; expected calibration error (ECE); ordinal mean absolute error (MAE) for score; span-level F1 for span; exact match and per-field accuracy for json. One metric is specific to this benchmark’s transliteration design: the accuracy gap between a native-script example and its transliterated pair. It tests directly whether a model’s Indic-language competence depends on script. That is a specific, testable version of the general script-sensitivity failure laya’s own README documents for its English checkpoint on non-Latin scripts (§1). Results should be reported per language, per language family (principle 4, §3), and pooled. The per-family breakout is the primary reported number, not the pooled average alone, so that strong Indo-Aryan performance cannot mask weak Dravidian performance or the reverse.

Any future evaluation of Maestro Flash, laya, or any other system on ITDB must state which split a number comes from: the core human-verified split or the extended machine-translated, lightly reviewed split. Given the explicit difference in verification level in §4.1, merging the two splits into one number would undermine the whole point of principle 2 (§3).

6. Limitations

This is a construction proposal, not a completed dataset. Specific risks and open questions:

  1. Native-speaker annotation at the scale a statistically meaningful benchmark needs is expensive. Laya’s own English benchmark uses 400 cases and 2,000 decisions per theme; matching that per language across 6 core languages implies a substantial annotation budget not yet costed in this paper.
  2. The extended-language machine-translated split, even with light review, risks the “translationese” problem the authors of MASSIVE and IndicXTREME explicitly designed around. Results on that split should be read cautiously and never presented without the core and extended distinction stated.
  3. Transliteration quality is itself a confound. An accuracy gap between native-script and transliterated variants could reflect either the model’s weakness or the transliteration tool’s errors. This paper does not yet propose a control for that confound beyond human spot-checks of a sample of transliterated output.
  4. Six core languages, while covering both major language families, still leave most of India’s 22 scheduled languages (and its many more non-scheduled languages) unaddressed. This proposal should be read as a first tranche, not a claim of comprehensive Indic coverage.

7. Conclusion

We propose ITDB, a human-verified, transliteration-aware typed-decision benchmark for Indic languages that covers every primitive. It is motivated by the documented Indic-language weaknesses of a currently shipping open system, laya, and by a specific gap in existing multilingual and Indic-specific benchmarks: none of MASSIVE, XNLI or IndicXTREME tests the choice, score, noul, span and json primitives this project’s systems use, and none includes transliterated or code-mixed input as a first-class variant. Once built, this dataset is the primary evaluation instrument that the Maestro Flash 90-day plan sets aside explicit time to construct and use.

References

  1. FitzGerald et al. “MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages.” arXiv:2204.08582
  2. Conneau et al. “XNLI: Evaluating Cross-lingual Sentence Representations.” arXiv:1809.05053
  3. “Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages” (IndicCorp / IndicXTREME). arXiv:2212.05409
  4. “IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages.” arXiv:2404.16816
  5. Khanuja et al. “MuRIL: Multilingual Representations for Indian Languages.” arXiv:2103.10730
  6. “IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages.” arXiv:2305.16307
  7. Casanueva, Temčinas, Gerz, Henderson, Vulić. “Efficient Intent Detection with Dual Sentence Encoders” (Banking77). arXiv:2003.04807
  8. “Paramanu: Compact and Competitive Monolingual Language Models for Low-Resource Morphologically Rich Indian Languages.” arXiv:2401.18034
  9. Laya (github.com/NandhaKishorM/laya), README.md and BENCHMARKS.md, read directly, 2026.

← All papers