Open pipeline expands Indic training data to 2.8 trillion tokens
NVIDIA, IIT Madras, and AI4Bharat are releasing 2.8 trillion tokens across 22 Indian languages, giving model builders a large new base for Indic pretraining and a recipe to generate more. The GPU-accelerated workflow translates curated English material such as math, code, and Q&A, then checks outputs against their source documents. It covers native scripts in all 22 languages plus romanized text for twelve, spanning 34 language-script combinations.