Generative AI (GenAI) has the potential to revolutionize clinical workflows, patient care, and accelerate medical research and drug discovery. As adoption of GenAI continues to grow, ensuring reliability, data security, and compliance is key to better healthcare outcomes.
To explore the opportunities and best practices for implementing GenAI, we sat down with Dr. Girish N. Nadkarni of the Icahn School of Medicine at Mount Sinai — informaticist, AI-driven precision medicine pioneer, and co-developer of the first FDA-approved AI bioprognostic tool, KidneyIntelX.
How GenAI is Making an Impact in Healthcare Today
GenAI is actively being deployed across healthcare systems, primarily aiding with automation of back-office or administrative tasks. Use cases where GenAI is making an impact today include:
- Back-Office Automation: GenAI is helping to automate routine back-office tasks such as appointment scheduling, billing, and insurance claims processing. At institutions like Mount Sinai, GenAI is improving financial management and streamlining patient registration processes.
- Ambient AI for Documentation: Physicians spend a significant portion of patient visits interacting with computers rather than patients. Generative AI-powered tools can provide transcription and summarization of doctor-patient interactions, allowing doctors to spend more time engaging with patients.
- Augmentation of Predictive AI: Predictive AI has long been used in healthcare for risk assessment, early disease detection, and more. Now, GenAI is building on these capabilities by providing contextual reasoning of predictive AI’s outputs and automating adjacent tasks. For example, a predictive model may flag patients at high risk of falls, while a generative model provides context about why the model flagged those patients and adds a note to the patients’ charts.
GenAI Use Cases that are Gaining the Most Traction
As we look ahead to the future, some of the areas GenAI is picking up the most traction include:
- Precision Medicine and Drug Discovery: AI has the potential to revolutionize personalized medicine by analyzing genetic, clinical, and environmental data to predict disease risk and tailor treatments to the patient. AI-driven drug discovery can accelerate the identification of new therapies, with the long-term vision of creating personalized medications for individual patients.
- Diagnostic Imaging: Advancements in AI are enhancing disease detection in radiology scans and bio prognostics. By analyzing complex imaging data with speed and accuracy, GenAI can improve early diagnosis and treatment planning.
- Patient Engagement and Education: Simplifying complex medical documents into patient-friendly language enhances consent processes and health literacy, ensuring clearer understanding of waivers, conditions, and treatments.
Areas Where GenAI Use in Healthcare Requires Caution
While genAI has great potential in healthcare, certain scenarios present ethical or operational concerns when GenAI is introduced. These use cases should be approached with caution:
- Patients Lacking Adequate Capacity: AI presents risks in cases where patients lack medical or legal capacity to make decisions, such as individuals with severe cognitive impairments.
- Pediatrics: GenAI use in pediatric care raises ethical concerns, particularly around informed consent and long-term effects.
- Protected Populations: Using GenAI enabled medical methods raises ethical concerns when treating protected populations such as prisoners or people with severe mental health issues.
Safeguarding LLM Applications from Hallucinations
One of the biggest technical concerns with GenAI is hallucinations — when LLMs generate incorrect or misleading information. Hallucinations can lead to serious consequences if left unchecked, especially if these models are making significant, irreversible decisions or recommendations. While hallucinations can never be fully eliminated, measures can be taken to reduce risks.
Techniques like Retrieval-Augmented Generation (RAG) help ground AI outputs by connecting them to verified medical knowledge and institutional policies.
LLM guardrails are another effective approach to moderate hallucinations by the LLM before they reach the end user. 3rd party guardrails act as an independent safety system, moderating each LLM prompt and response based on desired metrics – such as hallucinations, jailbreak attempts, and more.
A risk-based approach is crucial when deploying GenAI in healthcare. The severity and reversibility of a decision determine the level of oversight required. For non-critical, easily reversible decisions, AI integration and automation can be more aggressive. However, in high-stakes scenarios — such as AI-assisted diagnosis or treatment recommendations — strict output validation and human oversight remain essential.
Best Practices for Implementing GenAI in Healthcare
AI Governance frameworks are a great starting point for ensuring reliability, security, and compliance of GenAI deployments. With the correct checks and balances in place, healthcare institutions can feel confident about the safety and performance of their AI deployments. Key principles for building a healthcare AI Governance framework include:
- Clarity of Purpose: Implement AI with a well-defined objective, addressing a specific operational or clinical challenge.
- Risk-Based Oversight: AI applications and their outputs should be evaluated based on the criticality, reversibility, and urgency of the decisions they influence.
- Continuous Monitoring and Feedback: AI Observability provides ongoing model performance evaluations to proactively detect risks, analyze issues, and track key metrics.
- Training and Education: Healthcare professionals must be equipped with the knowledge to operate and oversee AI systems, and interpret their outputs effectively.
Keeping Patients at the Center
As healthcare leaders navigate the integration of this transformative technology, it is imperative to keep patient well-being at the core of decision-making. Generative AI is a powerful tool, but its deployment must be guided by safety, efficiency, and ethical considerations.
By establishing strong governance frameworks and continuously monitoring AI’s impact, the healthcare industry can harness GenAI’s potential while upholding the highest standards of patient care.
Watch the full AI Explained below:
