Transparency is Not Accountability: What Hospitals Can Learn from the Challenges of Open AI Models

Authored by Ihtimam Hossain, Head of Clinical Advisory, International Segment, Wolters Kluwer

Generative AI is rapidly transforming clinical decision support. Many tools can now retrieve literature, summarize evidence, and provide citations in seconds. While this creates the appearance of transparency, healthcare leaders must ask a deeper question: who is accountable for the way evidence is selected, interpreted, and presented?

As we embrace the undeniable power of generative AI to streamline clinical workflows, healthcare leadership must also confront three hidden risks within purely algorithmic models.

1. The Methodological Black Box: Looking Beyond Citations

Many medical AI tools provide citations, but transparency about sources is not the same as transparency about how evidence was selected and evaluated.

While users can see the studies cited, they often cannot determine why certain publications were prioritized, why others were excluded, or how conflicting evidence was weighed. Unlike traditional evidence reviews, these processes are rarely visible or reproducible.

Two important questions remain:

• The Search Strategy: How does the system determine which studies to retrieve from thousands of potentially relevant publications? In evidence-based medicine, search methodologies are explicit and subject to peer review. AI-driven retrieval processes are often less transparent to end users.
• The Evaluation Gap: Finding evidence is only the first step. Clinical decision-making requires assessing study quality, risk of bias, and strength of evidence. Frameworks such as GRADE make these criteria explicit, but many AI systems provide limited visibility into how evidence quality influences their responses. As a result, users may struggle to understand which studies carry the greatest weight and why.

Consider a clinician querying an AI tool for treatment options in a complex oncology case. The system may cite trials supporting a newly approved therapy while failing to surface smaller studies highlighting safety concerns. The presence of citations can create confidence, yet important context may still be missing.

2. The Commercial Blind Spot: Partnerships and Monetization

A less visible source of bias in medical AI may arise from the commercial structures that govern access to content and the incentives that shape platform development.

• Content Access Bias (Data Density Bias): AI systems can only analyze the content they are permitted to access. Differences in licensing agreements, content availability, or publisher partnerships may influence which evidence is retrieved and synthesized. As a result, certain sources may be surfaced more frequently than others, creating potential blind spots that are not readily apparent to users.
• Commercial Influence: Healthcare organizations should also understand whether business relationships could influence information prioritization. While responsible AI developers implement safeguards to protect clinical integrity, transparency around governance, content sourcing, and conflict-of-interest management remains essential. In healthcare, even the perception of commercial influence can affect trust in recommendations.

As AI adoption grows, healthcare leaders must evaluate both model performance and governance.

3. The Accountability Gap: Who Stands Behind the Answer?

This leads us to the most critical difference between AI-generated responses and human-authored clinical content: The most important distinction between AI-generated responses and traditional clinical content is accountability. While AI can retrieve and summarize information, responsibility for the interpretation and stewardship of that information is often less clear.

In a human-authored system, recommendations are supported by named experts, peer review, and established editorial processes. The methodology behind those recommendations is transparent, giving clinicians visibility into how evidence has been evaluated and synthesized.

In contrast, when an AI misinterprets a primary source, overlooks conflicting evidence, or presents an incomplete synthesis, the path from evidence retrieval to recommendation can be difficult to scrutinize. An incomplete or biased synthesis of evidence can contribute to unnecessary testing, delayed diagnoses, or inappropriate therapy choices. The challenge is not simply identifying that an error occurred, but understanding why it occurred, how it is reviewed, and what mechanisms exist to prevent it from recurring.

Moving from Search to Synthesis

It is important to acknowledge that AI developers are actively improving performance through techniques such as retrieval grounding and guardrails designed to reduce hallucinations and improve accuracy. These advances improve accuracy but do not fully address questions of governance and accountability.

For healthcare leaders, speed matters, but trust matters more. As organizations evaluate AI-enabled clinical decision solutions, they should ask a deeper question: Is the system simply retrieving information, or is it providing evidence-based synthesis supported by transparent methodology and clinical oversight?

Transparency is seeing the source. Accountability is understanding why that source was chosen, how the evidence was evaluated, and who stands behind the recommendation.

A Different Approach: UpToDate Expert AI

If accountability requires transparent methodology, rigorous evidence evaluation, and clear clinical ownership, then AI solutions must be designed differently from general-purpose retrieval systems. The future of clinical AI lies not only in faster search, but in trusted synthesis grounded in validated editorial processes.

For organizations seeking the speed of generative AI without sacrificing clinical rigor, UpToDate Expert AI combines advanced AI capabilities with UpToDate's expert-authored, peer-reviewed clinical content and established editorial standards.

Learn more about how UpToDate Expert AI combines the speed of AI with the safety and clinical rigor of UpToDate.

References

1. Li, J., Zhou, Z., Lyu, H., & Wang, Z. (2025). Large language models-powered clinical decision support: enhancing or replacing human expertise? Intelligent Medicine, 5, 1–4. https://doi.org/10.1016/j.imed.2025.01.001 Cited by: 52
2. Amugongo, L. M., Mascheroni, P., Brooks, S. G., Doering, S., & Seidel, J. (2024). Retrieval Augmented Generation for Large Language Models in Healthcare: A Systematic Review. Preprints. https://doi.org/10.20944/preprints202407.0876.v1 Cited by: 230
3. Liu, S., McCoy, A. B., & Wright, A. (2025). Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines. Journal of the American Medical Informatics Association, 32(4), 605. https://doi.org/10.1093/jamia/ocaf008 Cited by: 174

Disclaimer: This is an authored article; DHN is not liable for the claims made in the same.

Stay tuned for more such updates on Digital Health News