Eka.care Launches KARMA to Evaluate & Benchmark Clinical AI in India

The KARMA toolkit is designed as a reproducible testbed where developers can benchmark AI models on real, anonymized datasets.

Eka.care has launched an open-source AI evaluation framework called KARMA (Klarity AI and Radiology Model Assessment) to test and benchmark clinical artificial intelligence models developed for Indian healthcare.

The tool aims to bring transparency and standardized evaluation to the growing use of AI in medical diagnostics.

The KARMA toolkit is designed as a reproducible testbed where developers can benchmark AI models on real, anonymized datasets. Initially focused on chest X-rays, the tool allows clinical and technical experts to evaluate model performance across key metrics such as sensitivity, specificity, and explainability.

According to Vikalp Sahni, Co-founder and CEO of Eka.care, KARMA has been created to enable clinical teams to assess models better. With this, their aim is to bring standardization, transparency, and benchmarking into the Indian ecosystem.

The first dataset released as part of KARMA includes 5,000 anonymized chest X-rays from reputed Indian hospitals. The company said more datasets will be added in the future to expand its utility across other clinical specialties.

The KARMA framework is now available on GitHub for researchers, developers, and medical institutions to access and contribute.

More About KARMA

Short for Knowledge Assessment and Reasoning for Medical Applications, KARMA is built to evaluate medical AI systems across multiple modalities, including text, image, and audio, using 19 healthcare datasets and custom healthcare-specific metrics.

The platform is fully extendable, allowing researchers to plug in their own models and datasets via a registry system and even define custom evaluation metrics tailored to clinical needs such as speech recognition or multimodal diagnosis.

The framework comes with command-line and Python package access, supporting popular models like Qwen, MedGemma, IndicConformer, OpenAI, and AWS Bedrock. It also includes custom benchmarks such as Semantic WER and entity-based WER to assess the performance of medical ASR systems more meaningfully.

Eka.care’s launch includes four open datasets, curated and anonymized by a team of medical doctors:

• Medical ASR Evaluation Dataset
• Medical Records Parsing Evaluation Dataset
• Structured Clinical Note Generation Dataset
• Eka Medical Summarization Dataset

Alongside this, two domain-specific models, Parrotlet-a-en-5b for medical speech recognition and Parrotlet-v-lite-4b for clinical document understanding, have been released under an open MIT license.

Stay tuned for more such updates on Digital Health News