NVIDIA, IIT Madras and AI4Bharat announced Nemotron-pretraining-Indic-v1, a synthetic pretraining corpus of roughly 2.8 trillion tokens spanning the 22 Indian languages listed in India’s Constitution. The release will include the GPU-accelerated data generation pipeline used to produce the corpus, enabling developers to reproduce or adapt the workflow.
Corpus and language coverage
The retained corpus totals about 2,825.6 billion tokens: 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, measured with the Gemma-4 tokenizer. It covers 34 language-script configurations — native-script data for all 22 languages and romanized text for twelve languages. The ten languages provided only in native script include Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, and Sindhi; the twelve dual-script languages include Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, and Urdu.
Generation pipeline and quality checks
The pipeline is implemented with NeMo Data Designer and uses models such as Gemma-4-31B-it and, for some languages, Bodhan AI’s Indic Translate. The workflow translates full documents, distinguishes content to preserve (for example, code, equations and tables), assesses outputs against English sources with a rubric inspired by Multidimensional Quality Metrics (MQM), and repairs low-scoring outputs.
Accepted translations required a rubric score of at least 75 out of 100. Repair and reassessment recovered 438.4 billion tokens that are included in the published totals. For romanized variants, accepted native-script translations were converted to Latin script and rechecked against their native-script inputs to reduce hallucinated additions and residual English.
Model evaluation and how to use the data
The Nemotron 3.5 Super VL technical report documents a continued-pretraining (CPT) stage that used a configured budget of 100 billion tokens, allocating 10 billion Indic tokens (a mix of synthetic and natural text) to the model. Reported gains on five Indic benchmarks — MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy and Indic GSM8K — ranged from 4.98 to 9.82 percentage points after that CPT stage. The report notes a 2.04-point drop on IFEval 1.5 and describes improvements on IndiVibe dimensions for language quality; it also clarifies these results reflect the combined synthetic-and-natural CPT stage rather than isolating the corpus’s individual contribution.
When the corpus is available, users are advised to inspect dataset samples, review the dataset card and license, pin a dataset revision for reproducibility, and test mixtures at fixed token budgets while tracking per-language results. Configurations use language-script-region tags (for example, hi-Deva-IN for Hindi in Devanagari and hi-Latn-IN for romanized Hindi). The open Nemotron Bharat translation plugin for NeMo Data Designer provides prompts, evaluation routines and script-remapping support for adaptation to other source material.
Original source: Hugging Face Blog