US EHR Giant Epic Launches Its First Open-Source AI Validation Tool

Epic's open-source offering seeks to streamline this process by automating data collection and mapping, a major bottleneck for healthcare data scientists.

US electronic health records giant Epic Systems has launched its first-ever open-source software tool.

This freely available tool aims to empower healthcare organizations to validate and monitor AI models used within their systems.

The tool named seismometer has been made available on GitHub.

The software suite, dubbed "AI Trust and Assurance," addresses a critical challenge in the burgeoning field of healthcare AI.

Validating these models – essentially ensuring their accuracy, fairness, and safety – is a complex and time-consuming process.

Epic's open-source offering seeks to streamline this process by automating data collection and mapping, a major bottleneck for healthcare data scientists.

In April 2024, Epic announced that it would be rolling out the AI validation software soon. This is the first such open-source model launched by the EHR vendor.

Transparency & Collaboration Take Center Stage

Epic said making the tool available on GitHub, a popular open-source platform, is prioritizing transparency and collaboration.

This approach allows healthcare organizations of all sizes to access and utilize the software, fostering a more inclusive environment for AI development and validation in healthcare.

"By publishing on GitHub, it's truly available to everyone; it's not behind any lock or key that we control."It's fitting that a tool intended to ensure the equity of health AI is going to be publicly available and open to contributors from around the globe," said Corey Miller, VP of R&D at Epic told Fierce Healthcare.

Offerings of AI Trust & Assurance Suite

The AI Trust and Assurance suite offers several key advantages, Epic said. Healthcare organizations can validate AI models specific to their patient populations and workflows, ensuring their effectiveness within their unique environment.

The software allows users to analyze model performance across demographics such as age, race, and ethnicity. This functionality is crucial to mitigate bias and ensure models work fairly for all patients.

The automated data collection and reporting dashboards provide near real-time insights into model performance, facilitating ongoing monitoring and improvement.

Moreover, the open-source framework allows for the incorporation of evolving best practices and standards in AI validation as the field matures.

Epic said that the current version of the open-source tool does not validate the performance of generative AI models, but Epic plans to expand it to more AI models in the future.

Some stakeholders have expressed concerns about a potential conflict of interest, given Epic's dominant position in the healthcare IT market.

However, Epic maintains that the tool is agnostic to the source of the AI model and can be used to validate models developed by any vendor.

The company also plans to collaborate with healthcare organizations such as the Health AI Partnership (HAIP) and academic institutions such as the University of Wisconsin to refine the tool further and explore its capabilities.