The Translation Barrier
For decades, Indian languages have had to go through the indirect channel of English translation to interface with AI. With native Indian LLMs coming into the picture, a fundamental question arises: Do these models understand Native Indian languages better when prompted directly in those languages, or is translation still good enough? By pitting native language models against models like LLaMA and Mistral, the goal is to assess whether India truly needs native LLMs tailored for its languages or if existing, translation-dependent models can continue to meet the country’s diverse linguistic needs.
Methodology
To answer this question, we put together a complete evaluation framework for assessing these two processes —
Languages Chosen for Evaluation
Tamil, Telugu, and Hindi were chosen for evaluating the two approaches for several reasons. All the models support these three languages, which provides a consistent framework for comparison. Hindi is widely recognized and understood across large portions of India, presenting a strong representative of pan-Indian language usage. Tamil and Telugu are both linguistically rich and regionally distinct, providing interesting contrast. These languages also differ not only in script but also in syntactic structure, which allows us to see how the models handle varying features of languages.
Prompt Types for Testing
These four prompt types: -Q&A, summarization, coding, and instruction following were selected because they cover a wide variety of tasks that are common in the way language models are utilized in real life.
- Q&A — Tests the model’s ability to extract an accurate description of factual knowledge and present it.
- Summarization — Examines the ability to understand material and then condense it in a meaningful way.
- Coding — Tests the model’s ability for logical reasoning and to understand or create structured outputs.
- Instruction following — Tests the model’s ability to interpret and action human directives.
Together these four types give a well-rounded and real-world evaluation of a model’s capabilities both linguistically and functionally.
Models
Translated Prompt Models
Meta-Llama-3-8B and Mistral-7B-Instruct-v0.3 were selected to evaluate the translated prompt approach, as both models are mainly trained and optimized for English with limited cross-linguistic orientations, matching well with prompts that originate from translated Indian languages. Meta-Llama-3 is a powerful and ubiquitous general-purpose model, ensuring the standard for high-performance translation-based workflows, while Mistral-7B allows us to see how effective smaller, optimization models are in processing translated prompts in real-world contexts.
Native Language Models
Sarvam-m was selected for the native language evaluation as it is specifically trained to process and generate text in Indian languages like Tamil, Telugu, and Hindi, and was designed specifically to tackle Indic NLP tasks. Sarvam-M provides the ability to prompt directly in native languages, which was previously impossible without having to first translate the request into English.
Commonality Between the Three Models ?
Sarvam-M, Mistral-7B-Instruct-v0.3, and Meta-Llama-3–8B share a common family of architecture that makes them uniquely suited for comparisons. Sarvam-M is built on the Mistral architecture, which is derived from the Meta’s LLaMA model family.
sarvam-mis a multilingual, hybrid-reasoning, text-only language model built on Mistral-Small. (source- https://huggingface.co/)
This line of descent guarantees the three models have a similar transformer-based decoder-only architecture.This allows for consistency in structural capabilities which allows the evaluation to be focused solely on the differences in language handling (specifically the comparative effectiveness of native language prompting and translated prompting) without influencing the results through architectural inconsistency.
Model Comparison
Evaluation Metrics
- Latency — Translation Latency and Inference Latency
Prompt engineering latency is the total time it takes from a user inputting a prompt and then receiving the complete response from a language model. Prompt engineering latency has two main contributors to it: Translation latency and Inference latency . Understanding how long these two components take is important for any latency comparison of native and translated prompt workflows across multilingual settings.
Translation Latency
Translation latency pertains specifically to the translated prompt engineering type of approach. In this approach, the user’s prompt in its native language (e.g., Hindi, Tamil, Telugu, etc.) is translated first into English before the input is fed into an English driven LLM. In turn, the LLM will output in English, and may be translated back into the user’s original language. This two-step translation produces a significant latency governed by the length of the prompt, the complexity of the script, and the translations model’s performance (e.g., IndicTrans, Google Translate). Conversely, native prompt engineering completely avoids translation altogether. The prompts are submitted in the native language directly to models trained or fine-tuned for those languages.
Translated Prompt Engineering Workflow
Inference Latency
Inference latency is the amount of time that the language model takes to consider a given prompt, or a translated version of that prompt, and eventually respond. In native workflows, inference latency will vary with different inference latency drivers (less training data, tokenizing capabilities with complex scripts, and model size). On the other hand, in translated workflows, prominent English-language LLMs, such as Mistral or LLaMA, tend to have benefits from extensive-scale training and inference pipelines that can experience the best latency times during inference and response generation.
Native Prompt Engineering Workflow
2\. Cost
The cost metric in this evaluation is based on the total number of input and output tokens that were processed during model inference. Both the original prompt engineering and translated prompt engineering workflows are evaluated in this way, but translated prompt engineering will have additional tokens associated with it from the translation step. The cost is calculated using the calculated price modeling model from Hugging Face, which weighs costs based on the number of tokens processed per 1,000 tokens.
3\. Input vs Output Prompt Length
The input versus output prompt length is a decisive factor in evaluating the performance and cost of prompt engineering. Input length refers to the number of tokens in the prompt sent to the model whereas output length means the number of tokens produced as a result. Input and output length differs based on the task, language and prompting method.
Number of Runs
Each prompt type was run three times for each language in order to accommodate variability in model responses on top of any system-level factors (e.g. caching, load balancing etc.) that might impact results. This also ensured that the typical behavior of the model was captured .
Standardized Prompts
To facilitate fair and consistent evaluation across multiple large language models (LLMs), we utilize a set of standardized prompts that cover important task-types such as question answering, summarization, code generation, and instruction-following. The prompts were carefully translated into multiple languages to maintain meaning and difficulty, enabling evaluations and comparisons in multilingual and multitask contexts.
PROMPT\_TYPES = { "Q&A": { "hi": "ताज महल कहाँ है?", "ta": "தாஜ்மஹால் எங்கு உள்ளது?", "te": "తాజ్ మహల్ ఎక్కడ ఉంది?", }, "Summarization": { "hi": "स्वतंत्रता संग्राम का संक्षेप में वर्णन करें।", "ta": "இந்திய சுதந்திரப் போராட்டத்தின் சுருக்கம் என்ன?", "te": "భారత స్వాతంత్ర్య సమరంపై సంక్షిప్త వివరణ ఇవ్వండి.", }, "Code Generation": { "hi": "दो संख्याओं को जोड़ने वाला पायथन प्रोग्राम लिखें।", "ta": "இரு எண்களை கூட்டும் பைதான் நிரலை எழுதுங்கள்.", "te": "రెండు సంఖ్యలను కలిపే పైథాన్ ప్రోగ్రామ్ రాయండి.", }, "Instruction-following": { "hi": "चाय कैसे बनाएं? चरण दर चरण समझाएं।", "ta": "சாயை எப்படி தயாரிப்பது? படிப்படியாக விளக்கவும்.", "te": "టీ ఎలా తయారు చేయాలి? దశలవారీగా వివరించండి.", } }
Results
LlaMa
General Observations
- Inference latency was highly variable, even for identical prompts, largely due to backend factors such as caching and load balancing.
- There was no consistent correlation between token count and latency, suggesting that language complexity and internal processing play a significant role.
- In most cases, inference latency exceeded translation latency, making it the primary bottleneck in overall response time.
Language-Specific Trends
- Hindi exhibited the highest average inference latency (~74 seconds). This appears to be linked to longer English responses generated from translated prompts, which increase output token counts and thus processing time.
- Tamil occasionally showed unexpectedly high latency (e.g., one run exceeding 48 seconds) despite low output token counts. This suggests possible inefficiencies in processing scripts or syntactic complexity.
- Telugu had the lowest and most stable inference latency across runs, likely due to consistently shorter outputs that reduced the token generation burden.
Prompt-Type Specific Observations
- Q&A prompts resulted in moderate inference latency, as answers across languages tended to be concise. Output tokens remained within a similar range, leading to relatively stable response times.
- Summarization prompts showed wide variability in latency, driven by the open-ended nature of the task. In languages like Hindi, more detailed outputs may have contributed to longer inference times.
- Code Generation exhibited consistently low latency. Structured outputs and limited variability made this prompt type the most efficient in terms of processing time.
- Instruction-Following prompts also performed well, showing low latency and minimal variability. This can be attributed to the clearly bounded and structured nature of expected responses.
LLaMa Total Latency distribution in Translated Prompt Engineering
LlaMa Token Distribution in Translated Prompt Engineering
Llama Sum of Cost by Prompt Type in Translated Prompt Engineering
Mistral
- Inference latency varied across identical prompts, likely due to backend factors such as randomness or caching.
- Latency did not scale linearly with the number of tokens, indicating that internal processing complexity plays a larger role than output size alone.
- In most cases, inference latency was significantly higher than translation latency, establishing it as the primary performance bottleneck in the overall pipeline.
- Hindi exhibited the highest average inference latency (~3.9 seconds), likely due to the generation of longer English responses after translation.
- Tamil and Telugu consistently showed lower inference latencies (~700–800 milliseconds), suggesting more efficient handling of simpler or shorter prompts.
- Notably, even when output lengths were identical (e.g., 631 tokens), Hindi demonstrated greater latency variation, pointing to inefficiencies in processing or generation for translated content.
- Q&A prompts resulted in consistently low latency across all languages, benefiting from concise and predictable responses.
- Summarization prompts had slightly higher latency compared to Q&A but exhibited low variance, likely due to the uniform formatting and length of the outputs.
- Code Generation showed the lowest and most stable latency, with structured and compact outputs contributing to efficient inference.
- Instruction-Following prompts also maintained low latency with minimal variability, as the outputs were clearly scoped and bounded.
Mistral Total Latency distribution in Translated Prompt Engineering
Mistral Token Distribution in Translated Prompt Engineering
Mistral Sum of Cost by Prompt Type in Translated Prompt Engineering
Sarvam-M
- Inference latency showed noticeable inconsistency across identical prompts. For example, Hindi Q&A latency ranged from 1210 to 2041 milliseconds, indicating backend variability.
- Latency generally increased with higher output token counts, suggesting a direct relationship between response length and processing time.
- Compared to translation-based prompting, Sarvam-M demonstrated slightly lower inference latency overall. Since there is no need for additional translation steps, total response time was significantly reduced.
- Hindi had the highest inference latency, with summarization tasks reaching up to 6773 milliseconds. This is likely due to longer generated outputs and the added complexity of handling Devanagari script.
- Tamil showed unpredictable latency spikes indicating potential inefficiencies in processing Tamil script or syntax.
- Telugu was the most stable and consistently fast, especially for Q&A prompts (e.g., 1017 ms), likely due to shorter and structurally simpler outputs.
- Q&A tasks had moderate latency (1017–1524 ms) across all languages, with relatively consistent output lengths contributing to stable performance.
- Summarization showed the highest variability (5015–7134 ms), likely due to the open-ended nature of the task and differing language complexities.
- Code Generation consistently returned a Rate Limit Exceeded (Error 429), preventing successful execution across all runs.
- Instruction-Following prompts were handled efficiently (6335–7026 ms), with the bounded nature of the task reducing ambiguity and maintaining predictable output structures.
Sarvam-m Total Latency distribution in Native Prompt Engineering
Sarvam-m Token Distribution in Native Prompt Engineering
Sarvam-m Sum of Cost by Prompt Type in Native Prompt Engineering
Inferences
Latency
- Sarvam performs best overall and consistently shows the lowest latencies across most tasks and languages
- Mistral is second with moderate latencies, generally better than LLaMA
- LLaMA performs poorly with significantly higher latencies, especially for certain tasks give conclusion based on this
Token Distribution
- LLaMA is the most token-efficient with conservative output generation (35–325 tokens) and balanced input-to-output ratios.
- Sarvam shows moderate token usage with generous response generation (100–500 output tokens) while maintaining low input requirements.
- Mistral has the highest token consumption with extremely verbose responses (800–2200 output tokens) prioritizing comprehensive detail over efficiency.
Cost
LLaMA and Mistral are more cost effective options with all tasks under $0.002, making it the most affordable choice across all operations.
Savam has a moderate cost structure with expenses up to $0.0035, offering a balanced cost-performance ratio between affordability and capability
Agentic Workflow development for Native-Language Models
Agentic Workflow Development (Native-Language LLMs)
Agentic Workflows would allow AI agents to process native-language inputs intelligently and efficiently. This agentic workflow is one such example that intelligently chooses between native and fallback options while optimizing speed, cost, and accuracy.
The Agentic Workflow for Native-Language Models utilizes a user request in the user’s native language. The system identifies the language, assesses the user’s intent, looks for the tokens to see if it is entirely valid, evaluates if the prompt exceeds the token limit, and finally the assistant either summarizes or truncates the request if it does exceed the token limit and proceeds to processing.
This begins the next Agent selection and adaptation phase for the current request. The agent selects the ideal language model for the task and checks if a matching capable native-language model exists. If the native-language model exists, the native-language request will be adapted to the specific task; Summarization, Q&A, Classifications, etc. The task will be passed through the appropriate workflow.
Next is the Inference Execution phase where the agent will attempt to ‘do’ inference in the native-language. If it is successful, it moves on. If it is not successful, it will switch to fallback mode, translate the input to English, use and english LLM, and translate the output back into the user’s language, providing a guaranteed response if no native support exists.
The Output and Postprocessing stage outputs the final response in the native language, while also logging relevant metrics, such as total tokens used, response time, and model confidence, for future performance tracking and evaluation of the system.
The last module, Monitoring and Reporting, aggregates all the metrics accumulated throughout the previous stages, updates the available dashboards, and periodically reviews the outcomes to ensure the workflow remains efficient, equitable, and responsive to all supported languages and tasks.
The True Power of Native-Language Models
Language access should not determine local opportunity, security or well-being. Native-language AI models are changing lives by levelling the playing field and giving communities a chance to lift themselves up in their own voice. The following are some use cases that show the true power of native-language models.
Empowering Farmers with Local Knowledge — Now farmers can access real-time crop advice, pesticide use recommendations, and credit guidance ,all in their local dialects. This is not only a convenience, it is lifting livelihoods with the translation of life changing knowledge to those who need it most.
https://dr.lib.iastate.edu/server/api/core/bitstreams/f9dd5951-ec23-43b7-b5e2-f5cb77000ea6/content
Ethical AI Built by the People, for the People — Organizations like Karya are making sure that AI understands native languages, and is actually training it with native speakers. By paying local workers for their voice data in Kannada, Marathi, etc, they are making AI more accessible, accurate and authentic.
Mental Health Support Without Language Barriers — It’s hard to ask for help, let alone when it’s not even in your language. Native language LLMs can provide culturally sensitive, private, and trustworthy mental health support to people who might otherwise be ignored.
Accessible Legal Support Through Native Languages — Native-language LLMs can help individuals understand legal documents by providing clear explanations in the person’s primary language, making legal support more equitable.
Inclusive Disaster Communication and Emergency Response — Native-language LLMs can offer timely emergency alerts, safety instruction and support resource information in people’s languages in the event of a natural disaster, ensuring that life-saving information reaches everyone.
References
https://arxiv.org/html/2409.07054v2 https://arxiv.org/abs/2505.11665 https://arxiv.org/abs/2411.04025 https://link.springer.com/chapter/10.1007/978-3-031-75164-6\_4 https://dr.lib.iastate.edu/server/api/core/bitstreams/f9dd5951-ec23-43b7-b5e2-f5cb77000ea6/content https://time.com/6297403/the-workers-behind-ai-rarely-see-its-rewards-this-indian-startup-wants-to-fix-that/