
Entries in AI (6)
Sutter Health uses AI & GenAI to Personalize and Streamline Care Between Patients and Providers

Since 2023, Sutter has been addressing the provider burnout epidemic and the burdensome administrative tasks detracting from the time and focus clinicians dedicate to patients. Dr. Veena Jones, Chief Medial Information Officer, Sutter Health explains "by leveraging generative AI, we’re reducing these burdens, allowing clinicians to focus on meaningful, personalized interactions with their patients while enhancing the overall care experience and the clinicians’ joy of work. It’s truly a win-win for both providers and patients.”
Sutter Health, a non-profit integrated delivery health system providing comprehensive care to nearly 3.5 million patients throughout California, is one of the early health systems thoughtfully exploring ways to test AI-driven solutions to support patient and provider needs. With a focus on responsible implementation, Sutter is committed to leveraging AI in a way that enhances care while maintaining the highest standards of security and trust.
Specifically, Sutter has invested in AI & GenAI for two key initiatives as “tools for our providers tool kits”:
1) Supporting Patient Messaging within Epic, Augmented Response Technology (ART) – The ART tool analyzes patient messages, gathers relevant information and drafts a response for the clinician to review, edit and approve, before sending to patients. Integrated within Epic, ART uses a large language model (LLM) to generate a contextually relevant response for clinician review.
In Q4 2023, Sutter was one of the first handful of health systems that began piloting this tool.“Our early work with Epic and other leading health systems to pilot the ART tool was instrumental in advancing its development,” said Dr. Jones. “Since October 2023, ART has been in production within the Sutter EHR, following a collaborative effort with primary care clinician champions and patient advisors to help validate and refine its capabilities.”
Sutter started testing ART with about 70 physicians and advanced practice clinicians to gather feedback on messages generated from the LLM and adjust the prompts before the March 2024 launch. Beyond clinician feedback, the team collaborated with patients for feedback on GenAI generated message response.
"It was vital to include perspectives from patients in our Patient Family Advisory Council (PFAC) program," said Dr. Jones. "By reviewing message responses with patients, we gained invaluable insights—ensuring that the technology supports clear, compassionate, and meaningful communication. Patients are at the heart of our mission and it’s essential to align AI advancements with the human-centered care we strive to deliver.”
In March 2024, Sutter launched ART to all primary care clinicians in Internal Medicine and Family Medicine, since they were on the receiving end of the most patient inbox messages. Soon after, ART was made available to all pediatricians and then was expanded to OB/Gyn and geriatric clinicians across the organization.
Sutter recognized the need to adjust the LLM prompt to ensure relevance and benefit with each specialty rollout, working closely to test ART before making it more widely available. By the end of 2024, all Sutter clinicians receiving patient messages in their in basket had ART-generated draft replies available for their use.
AI PATIENT MESSAGING SUCCESS:
“We started the ART project with the goal of saving providers time when responding to patient emails. What we heard overwhelmingly was that providers want to keep using it because it significantly reduces their cognitive burden”, shares Laura Wilt, Sutter’s Chief Digital Officer. “Think about the level of effort required for a doctor to create a patient response email from scratch versus simply reviewing and editing a pre-drafted message.”
· To date, 3,000+ Sutter clinicians have used ART to assist with responding to medical advice requests from patients.
· With over 217,000 messages sent, feedback from Sutter clinicians has been invaluable in refining the prompt engineering process to enhance the quality of draft responses.
· Clinicians are reporting an average time savings of up to 20 percent when using ART to help manage their patient communications, along with a noticeable reduction in cognitive load and mental fatigue.
Qualitative Provider Feedback:
“I like the empathy and the AI component that reviews the chart and gathers information like last labs, or pending labs. That is very helpful.”
“I like that the responses are often more thorough and considerate than I would have time to write.”
“Includes empathy statements and acknowledgement of patient's symptoms. Really good for quick and straightforward questions.”
“Appreciate the ART-generated responses to long patient messages. Reduces mental workload by having a template initially started.”
2) Ambient Listening for Visit & Follow- up with Patient Friendly Summaries. Sutter has been working closely with Abridge which has a generative artificial intelligence (AI) platform for clinical documentation. It records and transcribes a visit and generates components of the clinical note. In addition it creates a patient-friendly summary of the visit which the provider can paste into their After Visit Summary.
To date, most clinicians using Abridge have seen a reduction in documentation time and cognitive burden associated with their work. The ambient listening technology allows for more personal connection and focus on the patient interaction, rather than on the computer, thus strengthening the care relationship.
“Another benefit we are seeing with Abridge’s ambient listening technology to personalize care is enabling providers to conduct the visit in the patient’s native language, while the English documentation is created in real-time. We have patients and clinicians who speak a variety of languages, from Spanish to Cantonese. The mental load is high for these clinicians having to translate everything they discuss into English for the note. Abridge takes care of the translation and puts the note directly into the EMR”, explains Wilt.
From April- December, Sutter enabled 1,000+ ambulatory physicians and advanced pratice clincian users with the ambient technology. Sutter learned about the importance of physicians being able to directly access Abridge through Epic’s Haiku app, having it fully integrated to make it easier to have their work all in one place. Wilt adds “we took a thoughtful approach to implementation, aiming for a smooth adoption. Our Digital Academy Team played a key role in supporting the rollout and ensuring physicians had the resources they needed. It went smoothly, with minimal training needed— just a brief eLearning video to watch. From a physician adoption perspective, it took off like a rocket ship… maybe because it takes tasks away and lets providers focus more on connecting with patients.”
AI AMBIENT LISTENING SUCCESS:
“Again, we initially focused on time savings, and for many clinicians it’s a significant time saver. But we’ve found that —it also helps reduce cognitive burden”, Wilt adds.
Qualitative Provider Feedback:
“Abridge has been a life-changing experience. It alleviates 50% of the charting work for me,” said Alice Woo, M.D., a plastic surgeon with Sutter West Bay Medical Group. “[Now] I feel like my conversations with patients are much more intimate and therapeutic.”
“Abridge has changed how I feel at the end of the day,“ said Kevin Chen, M.D., a family medicine physician with Palo Alto Foundation Medical Group. “If you have balance in your life, you can do other things. It makes you a better doctor when you’re in the clinic seeing patients—you’re better centered.”
EXPANDING AI USE CASES:
“We see the patient message response/ART in Epic and Abridge Ambient listening platform as examples of how generative AI technology can help Sutter personalize medicine”, Wilt shares.
Patient Message in EPIC/ART. Sutter is interested in moving beyond the patient generated messages into other message types such as “results” (i.e. normal results) and “medication” (i.e. medication refill requests). Sutter is also planning a pilot with Epic for another feature to help further streamline physician workflows with “Chart Summarization”, which provides a summary of key patient information within their electronic medical record.
Ambient Listening/Abridge: Sutter is extending use of the ambient listening tool in a pilot with emergency and inpatient physicians across all Sutter hospitals. They see opportunities with other types of clinical staff, such as with nurses and physical therapists, and look forward to exploring these as the technology evolves.
Sutter is excited about the continued evolution of Abridge which will "go beyond capturing the conversation and generating a note, to queuing up clinical orders that were discussed. Wilt explains, “so if your doctor mentions getting an Xray, the order would be put in for the physician to just accept, instead of having to type in the order.”
“AI is all about continuous improvement—what was implemented a few months ago has already evolved, and it will continue to improve over time," said Dr. Jones. She encourages Sutter teams and other health systems to approach these tools with curiosity. "When implemented thoughtfully, AI has the potential to not only reduce administrative burdens but also strengthen the connection between providers and patients, enabling more meaningful care experiences."
MY TAKE: AI INNOVATION BEST PRACTICES
Sutter has invested in the right practices and processes to drive success:
- Collaborated closely and early with vendors (e.g. Epic, Abridge) to define and pilot AI and GenAI capabilities.
- Co-created with key clinical and patient stakeholders to ensure a strong experience.
- Integrated with exisiting workflows spanning development, training, care delviery and support.
- Focused on both qualitative and quantitative success measures including the voice of the clinician and patient (e.g. feedback) and user trial/usage/traction.

Providence Health Supercharges Primary Care with AI for Specialist Care Decisions

During the HIMSS AI for Healthcare Forum in Boston, Dr. Eve Cunningham, MD, MBA, GVP, Chief of Virtual Care and Digital Health at Providence Health presented a few AI use cases including an impactful example which empowers PCPs for better care decisions.
Today primary care doctors are pressured to address patient health concerns in an average of 18 minutes with an average of 220,000 data points in the patient’s electronic medical record. With over one in 3 patients referred to a specialist each year, PCPs are tasked with diagnosing the health issue to and making decisions about the need for a specialist.
Barriers to Specialist Care Decision
With vast amounts of patient data that is not clinically organized around context within the patient’s chart, PCPs must spend time pulling together and reviewing relevant information to support the patient’s health problem. Then the PCP must determine if a specialist is needed, and if so which type of specialist.
Given the shortage of primary care doctors and specialists, optimization of the referral process is critical to access to care. However, the referral process today is both inefficient and ineffective causing problems:
- Patient did not need to go to a specialist. (Note: About 20% of patients referred to specialists do not need to see a specialist at all)
- Patient is sent to wrong specialist and/or at inappropriate level of urgency.
- Patient arrives to specialist without any prereferral work-up such as lab, imaging and first-line therapy for a condition.
- Patient has delayed care until there is a specialist appointment
Accelerating Primary Care Referrals
The September issue of NEJM Catalyst’s article on “A Physician-Created Platform to Speed Clinical Decision-Making and Referral Workflow” describes the problem and process that Providence Health has undertaken to conceptualize and build their MedPearl Platform.
“As primary care clinicians struggle to access the expanding base of specialized knowledge and experiences, they attempt to refer patients to specialists, often without the advantage of usable tools at the point of care.”
Developing this needed platform entailed using a “Human- Centered Design approach with a Jobs to Be Done framework, working closely with our clinicians and UX designers”, explains Dr. Cunningham. “We began by asking clinicians how this platform needed to work and look, and got input from 270 clinicians in the design process, which was essential for gaining buy-in”.
MedPearl Platform Built by & for Clinical Care Decisions
Based on clinician’s requirements, Providence built MedPearl as an application within their Epic electronic medical record (EMR) to integrate a digital assistant with clinical knowledge to support primary and urgent care clinicians. MedPearl has “concise, human-authored algorithmic primary and subspecialty care next best actions on over 700 conditions”.
Clinicians utilize MedPearl’s AI algorithms to pull up and review essential clinical information alongside contextualized patient data to see next best actions with what has/hasn’t been done for the patient, empowering clinicians to optimize the “work up” while reducing redundant diagnostics.
Since it can take weeks to months for the patient to get in to see the specialist, the MedPearl platform enables PCPs and urgent care providers to take care of the diagnostics in the interim to optimize the care referral decision. With MedPearl, clinicians can see the testing stream, make sure tests do not need a referral, find the most appropriate specialists and triage the urgency of the specialist appointment based on patient history and diagnostics. Think about what this means for rural health where accessing a specialist is even more challenging.
Providence has designed MedPearl with icons to reduce the clinicians cognitive reading load and provides links to guidelines and QR codes to scan for evidence -based applications.
Clinician Experience with MedPearl
Providence Health has deployed MedPearl across their health system which is available in Epic’s Marketplace.
Dr Cunningham as MedPearl founder comments “Our clinicians tell us that with MedPearl they are spending less time in the EMR, fewer outside work hours and are more productive. To date, we have 7,000 clinician users, making 30,000 searches per month on the MedPearl platform”.
During the initial pilot with 216 participants conducting 14,000 searches, clinicians reported the clinical content helped with clinical decision support: they did not need to refer the patient 20% of time, improved care plan and work-up 72% of the time and changed referral specialty and/or level of urgency 20% of the time.
With MedPearl, clinicians are also realizing better resource efficiency. “We had a challenge with some of our more newly onboarding clinicians overwhelming our Medical Directors with clinical questions and curbside phone calls. The call volume decreased significantly when these clinicians were advised to reference MedPearl first for answers”, shares Dr. Cunningham.
Clinicians often walk through the MedPearl tool with their patients. They review educational content together, discuss the algorithms and use the decision support tools.
Feedback from Clinicans Using MedPearl
After using the MedPearl tool, clinicians commented about the ease of use, quick access to the most essential patient data and benefits of leveraging this patient data for the physician, practice and patient.
“MedPearl provides concise and focused clinical evidence -based summaries, making it easier to quickly access the most relevant information. Its streamlined format saves time and effort, ensuring health professionals can find information without being overwhelmed by unnecessary details”. Medical Director Urgent Care
“The cognitive offloading and reassurance that knowing the Prov approved standard of care has been helpful for me, and I love being able to share it with newer clinicians”. Urgent Care Physician
“The Headache algorithm – perfect example of how to reduce variation in our urgent care practice.” Urgent Care Physician
“With the demand for neurology consultation greatly exceeding the supply of neurologists for the foreseeable future, MedPearl improves access when it has never been more vital.” Chief of Neurology
“MedPearl is a point of care tool we need to augment our clinical acumen and to help drive uniformity of patient care path the best -practice level. This allows primary care physicians to provide more care value, improving diagnostic testing, treatment, and referrals and in opening more specialist access.” Medical Group Director
“Recommendations re: referrals and specific imaging with and without contrast very helpful. Truly has improved physician satisfaction and flow for the day”. Family Medicine Physician
“Contextualization is pretty cool. Makes Pearl useful in multiple ways.. Quick way to see all relevant data about a work up for a particular topic.” Internal Medicine Specialist
“Love the patient experience, engagement and shared decision-making components of this tool.” Practice Senior Director of Operations
“I love the headache algorithm. During a recent appointment, I had a patient point to each symptom that applied to her.” Physician
Future Direction of MedPearl for Health Systems
Dr. Cunningham describes Providence Health’s approach with “feedback and iteration as a core principle by which we drive our MedPearl features roadmap and design. We also push our end users to think beyond the workflow of today, and envision workflow of tomorrow.”
The MedPearl team continues to collaborate with users to gain insight into future enhancements. Specifically, they are focused on these platform innovations:
1. "Expanding contextualization beyond labs, imaging and procedures to further enrich the context aware data aggregation experience"
2. "Adding documentation support features to expedite documentation workflow"
3. "Further refine home page to create a more dynamic experience to delight end users"
4. “Since we believe all workflows will converge on ambient technology, we are preparing ourselves to integrate into ambient workflows. We don’t plan to be the ambient provider, rather we envision ourselves as a clinical intelligence plug-in to further enrich that experience.”
Dr. Cunningham confirms plans to begin “deploying MedPearl to other health systems using EPIC and then Oracle EMRs. We see ourselves as ‘headless’ clinical decision support system that is EMR agnostic and application agnostic. We want to permeate and spread the goodness of MedPearl in a seamless fashion, and meet clinicians and patients where they are“, Dr. Cunningham concludes.
Providence Saint John uses AI platform to Empower Cancer Patients & Providers for Better Care

In October 2022, Providence Saint John's Health Center invested in Project Ronin, an AI driven cancer intelligence platform to empower cancer patients and their clinical teams to manage treatment symptoms, care communications and clinical decision support. Their goal is to reduce ED visits and deliver better patient care and outcomes.
According to the American Cancer Society’s 2023 Cancer Statistics report, cancer remains the second highest cause of death after heart disease. This is an estimated 2 million newly diagnosed cancer cases resulting in about 610,000 deaths.
Dr. Martin, MD, CMO of Providence Saint John’s believes that providing a new “space where cancer people can engage with their physicians has some very real, tangible benefits.” The digital tool is designed to support “patients with less-critical symptoms, which has the potential to reduce emergency department visits and improve clinical efficiency, which means a more efficient utilization of resources. Of course, we always advocate patients to come to the emergency department if they believe they are experiencing a medical emergency.”
Cancer Care Collaboration
Successful treatment of cancer is multifaceted, requires a multi-disciplinary care team, entails managing different data sets with continuous capture and monitoring of data, and has data complexity given the non-linear nature of the disease (i.e. diagnosis, treatment, remission, recurrence, treatment, survivorship/death).
Patients struggle to manage their cancer, both mentally and physically. Patient treatment symptoms can be severe. Questions come up between treatments but care teams are extremely busy. Patients need to remain connected to care teams to stay engaged, better manage, and communicate side effects.
Providers try to navigate their inefficient clinical workflows, address the administrative burden due to documentation requirements, and deal with workforce shortages. Today, providers spend 20-40 minutes finding patient info to prepare for patient visits, sometimes missing key data. Providers are overwhelmed to make needed decisions for cancer patients at the point of care.
AI Cancer Platform Delivers Care & Decision Support
The Ronin Cancer Platform enables patients to “take control of their care in a way that wasn't possible before and feel better connected to their care teams,” explains Dr. Neil Martin, MD, CMO of Providence Southern California Clinical Institutes and Executive Director of Saint John's Cancer Institute. Clinicians utilize this cancer intelligence platform’s predictive analytics and clinical dashboard to define treatment pathways and support more informed care decisions.
How does this work? The Ronin cancer platform takes structured and unstructured data from different EHR silos, and then cleans, calibrates, contextualizes this information to present care insights. With predictive analytics, providers can identify at-risk patients and see a complete real-time view of the care journey for immediate actions.
AI Data- Driven Cancer Platform: Patient Experience
Patient Patricia (patient persona), 74 years old, is newly diagnosed with breast cancer. While meeting with her new care team, Patricia learns about new resources including the free Ronin Symptom Monitoring App, to educate and support her as she navigates her cancer path. During her next appointment, Patricia notices the poster in her doctor’s office prompting her to download this patient app.
Patient Symptom Monitoring App Poster:
We want to support you through your treatment. In between your visits, please let us know how you're feeling by using the Ronin Symptom Monitoring Application. This will help us catch any side effects that you may experience. Studies show that symptom monitoring can ...
• Help people live longer and improve their quality of life
• Help people stay on treatment longer
• Lower visits to the emergency room and reduce hospital stay
Patricia downloads and begins using the Symptom Monitoring app which captures her Patient Reported Outcomes (PROs), enables her to track and manage side effects for treatment tolerance, displays her symptom history and provides her with personalized education based on what she is reporting. Within the app, Patricia selects from a list of symptoms and then sees a message that her symptoms “are concerning”, that her care team is monitoring this and will reach out to her within 24 hours.
AI Data-Driven Cancer Platform: Provider Experience
Patricia’s care team receives a “symptom alert” through the EHR based on the information that she has entered. Her symptom information has required a response (e.g. new symptom, moderate or severe symptom). The Nurse Navigator reaches out electronically or telephonically to check in on Patricia to understand more about her symptoms.
When her Oncologist Dr. Davis initially set up Patricia’s personalized care pathway on the platform, he indicated her specific disease type, which triggered patient reported outcomes surveys and at-risk surveys which captures data and intelligently prioritizes for the care team to take quick action to course correct and preempt adverse events.
While preparing for an upcoming office visit with Patricia, Dr. Davis uses the Ronin Platform to view a longitudinal timeline of her cancer journey, filled with real-time data (e.g. labs, images, pathology, treatment, surgery notes, past encounters) to make informed, proactive and personalized treatment decisions. Following the visit, Dr. Davis (or the clinical staff) pushes educational content to help Patricia to navigate her new symptoms or cancer stages.
AI Driven Care Experience Value Proposition
At Providence Saint John's Health Center, both patients and providers are finding value using the AI Ronin Platform, including clinical efficiencies, better care and outcomes and higher satisfaction. Here are some comments from the patient and provider users:
Patient Experience:
“Ronin provided confidence someone was monitoring me.”
“Ronin made me feel like my care team was with me, even at home.”
“I prefer not having to call the hospital. (With app) I am able to record side-effects right away as they are happening”.
“It’s (app) so easy to open up and write how I’m feeling…It’s a great reminder to me that even though I may think I’m fine, I need to pay attention to my symptoms.”
Provider/Staff Experience:
“We were able to catch an immune- related toxicity early for a patient of mine. Without Ronin, I would not have caught it”. Medical Oncologist
“We want to make sure the patients are having the best quality of life. With Ronin, we can check in on them more frequently.” Nurse Navigator
“From the treatment team’s standpoint, it’s made it easy to follow patients. It’s a great way to stay hands-on.” Nurse Navigator
“I had a patient straight up tell me ‘If it wasn’t for the app, I would be calling you guys every day'.” Medical Oncologist
“Visually, the Timeline tab makes it much easier to get up to speed on a patient. Especially compared to reading my colleague’s last notes.” Medical Oncologist
Success Metrics
In addition to gathering qualitative comments about the patient and provider/staff experience, Providence Saint John’s tracks quantitative success metrics for the AI driven Cancer Program. Some key metrics include:
● Reduced unnecessary ED admissions by >20% (early findings)
● Improved efficiencies with 35% decreased patient call volume and 32% patient-initiated messages.
● Strong patient satisfaction & experience
- 84% see positive impact on care experience
- 90% say app is easy to use
- 88% are satisfied with solution
Future for Providence with Ronin Cancer Platform
Moving beyond the patient mobile app and care communication from phase one, Providence Saint John's is planning to provide clinical decision support with oncology and risk dashboards and the capability to view comparative patient analytics in phase two.
"We are proud to implement evidence-based care solutions for patients in our community," said Brad Bott, Executive Director of the Southern California Clinical Institutes. "Our partnership with Ronin will help improve cancer care delivery while empowering our staff to practice more efficiently. It's a win-win."
Providence Health’s Bot Grace engages and guides patients to the digital front door for care

According to the 2019 Accenture Digital Health Consumer Survey, patient’s expectations are increasing for providers to offer digital capabilities. Patients value convenience (appointment times, location) and are increasingly considering “non-traditional” service channels such as retail clinics and virtual care.
Providence Health’s Innovation team continuously monitors these trends to identify ways to deliver a better patient care experience. “With the tremendous growth of chatbots and voice assistants, we decided to develop a virtual health assistant and begin to understand how to support the patients in their search for care”, explains Maryam Gholami, Chief Product Officer at Digital Innovations, Providence Health.
In fall 2018, Washington State based Providence St Joseph started developing and testing Grace, a chatbot which asks the patient about their symptoms and based on their location, time of the day and scope of services provided in various care options (virtual, clinics, or At Home) directs them to the best care. Eventually Grace was enhanced to also address frequently asked questions by patients.
Since the healthcare system is complex for consumers to navigate, Providence Health designed their digital tool to provide answers and guidance to help the patient find the right care and make the best care decisions. For example, a consumer may not know which modality of care to use (e.g. Emergency Room. Urgent Care, Retail Clinic, Virtual/Telehealth) or what symptoms trigger an immediate visit.
On the consumer side, Providence Health wants to stay aligned with their patient’s increasing expectations for service on demand. Consumers are already using digital assistants for shopping, banking and travel 24x7.
“From a business perspective, we want to provide accessible, convenient and affordable care. Therefore, we need to improve operational efficiency, free-up professional caregiver capacity for the right care and tasks, while increasing the care options and quality for consumers. We want to have our virtual health assistant take on the administrative tasks so that our clinical staff can focus on patient care. For example, Grace conducts a virtual patient intake to schedule a visit, collecting information such as reason for the visit, demographics, insurance and payment details”, explains Maryam. “This digital health assistant also assesses the best modality of care based on patient’s input and availability of care, matching the right resource to meet each patient’s needs. With this type of service navigation, Providence Health can ensure that the required license level is designated for the visit, reducing care cost and ensuring availability of providers for the appropriate level of care. Healthcare is supply-constrained, so this is a very important value.”
Patient Experience with Grace Bot
Providence Health has worked closely with the patients to design the Grace bot. Here is a look into the patient experience.
Patient Patricia who lives in Washington State, types her symptoms into Google to figure out what she has and where she can get care. Within her search results, Patricia sees a Providence Health website link and then meets Grace, a pop- up text chatbot ready to help her.
The Grace bot helps Patricia in two important ways:
1. Care Navigation: Given that Patricia needs “same day care” and has a low acuity health problem, she types into the Grace bot that she has a fever and sees that Grace has recommended a virtual visit or a same day clinic appointment. Patricia provides her zip code, chooses the clinic near her house and schedules an appointment for 7 PM that evening.
2. Customer Service/Concierge: Patricia has questions about whether her insurance is accepted and types in “what insurance do you take?” Grace provides a list of insurance options. Patricia can also ask for information to read about her health problem before her visit.
Patient Experience with Grace Bot
The Grace bot was designed for patients by patients. Throughout the development process, the product team at Providence Health worked closely with patients to understand the “tasks they were trying to complete” and defined how Grace needed to guide them through these steps. During testing, the team monitored where patients dropped off and tuned the messages to “optimize” their care journey.
“We are digging into the questions that our customers ask so that we can improve our knowledge base. This enables Grace to better respond to our patient questions”, Maryam explains.
With ongoing patient feedback, Providence Health has identified and has been addressing several areas to improve the patient experience.
Chatbot Persona: Maryam shares that “we used emojis to make the digital interaction more casual and friendly. Our patients did not like the emojis because they felt emojis do not convey trust which is essential when communicating health information. Patients also needed to feel comfortable that the responses and recommendations were not coming from a human. We are working on the right persona for Grace and defining the best interaction”.
Chatbot Behavior: Providence Health’s product team is determining when Grace needs to confirm information with the patient so that it accurately reflects her needs. There is a fine line between repeating back to be sure and being annoying to the patient who is looking to quickly complete her task (e.g. set up a visit). However, there is a risk of sending the patient to a modality that does not meet her needs (i.e. she needs to see a doctor and not an RN). “We need to understand the clinical operations from end to end so that we can define the business rules to successfully guide the patient interaction”, Maryam comments.
Consumer Adoption: Some patients are pleasantly surprised about what the Grace chatbot can do for them. Others may not understand what to ask her. Health organizations will need to educate consumers on the types of questions to ask so that patients can gain the most value when engaging with the bot.
Provider Adoption: Providence Health is working on enabling the Grace bot to support virtual care. Grace can engage with the patient to capture information prior to the telehealth visit. However, we need to ensure that this is designed and implemented in such a way that clinicians can properly review all of the information captured before the visit begins and that we are not increasing provider burn-out. It needs to improve the experience for both patients and providers.
Patient Representative Experience: When the Grace bot “hands off” the patient for the scheduled visit or for further follow up, it is important that the representative knows that patient just engaged with the Grace chatbot, has the context and the information already collected and can address the patient accordingly.
Chatbot Success Evaluation
Maryam shares some key success measures for the Grace bot. “We currently know that over 42% of our patients who engage with Grace get answers to their questions or get help navigating to the right modality of care. 18% of these engagements result in completion of appointments booked with one of our ExpressCare Clinics (Urgent Care). We are also learning which questions Grace doesn’t know how to answer or which tasks she can’t currently complete. This learning is very important to help us with our future product direction and iterations”.
Patients interacting with Grace can navigate to Providence Health’s ExpressCare options based on a chief complaint or use a symptom checker before choosing recommended ExpressCare options. Providence Health is seeing a 90% accuracy rate for patients that have been directed to ExpressCare from the patient’s chief complaint.
According to Maryam, there are weekly meetings with the product analytics team to understand how patients are using Grace so that they can enhance the patient experience. “We measure every interaction in the funnel to understand if we’ve helped users complete their tasks. Some of these measure include # of click rates, # of patients questions answered, # of appointments booked, # of visits completed along with type of visits, and chatbot engagement when the customer support center is closed”, describes Maryam.
At the end of each interaction, Grace asks the patient if he is satisfied with the task – “Was I able to address your needs today”? According to Maryam, about 40% indicate that the Grace bot met their needs. Providence Health is closely listening to patient feedback to increase the satisfaction percentage.
Providence Health has received insightful patient feedback about the Grace bot experience:
“This was the first time I’ve seen something like that, I thought it was extremely helpful. It asked a lot of good questions to get me the right help.”
“I felt like it was really easy to get an appointment. Very quick automated responses got me to the right place.”
“I thought it was excellent. I research a lot of my own health issues, so I’m excited about this. Anything that can help me learn about my symptoms and what to do about them, I’m excited for it.”
“For me, it was kind of cold. If it could change the wording to make it more human it would be better. I’d prefer a more human touch to it. I think in this field personality is important, so work on making it more warm, caring, and friendly.”
Future Plans with Grace:
Providence Health is committed to enhance the Grace virtual health assistant experience for patients by:
Expanding the patient journey: Currently, Grace is available more in the discovery phase, when patients are searching for care options. “Next, we are incorporating Grace into the Virtual waiting room to gather intake information before the patient participates in the telehealth visit. Our goal is to have Grace available when/where appropriate to help consumers navigation through health journey”, Maryam adds. “We are looking to move beyond low acuity to helping patients find and book PCP and Specialty appointments”.
Increasing the personalization: While Grace currently does some personalization based on consumer’s location, time and symptoms. Providence Health believes there is still a great deal of opportunity to leverage various sources of data including patient records. They will proceed carefully given the consumers’ data security and privacy needs.
Improving the High Tech & High Touch Collaboration: “We view our virtual assistant as an augmentation tool for our professional caregivers. We are working on creating a warm hand off to human caregivers with the conversation context. We are designing these tools so that based on consumer’s choice, human caregivers can step in to assist our patients”, explains Maryam.
Providence Health’s clinical and business leaders believe that offering these virtual health assistants are essential to delivering the best care experience for the patient, today and in the future.
“We are levering technology to connect the consumers to the right place for care with the right service and providers at the right time. AI is going to help us to improve the seamless experience.” Thanh Nguyen, Executive Director and Chief Clinical Officer of Express Care
“AI-powered virtual assistants will have a significant impact on healthcare by engaging consumers through voice/chat, a more natural way of interaction, and by intelligently automating mundane tasks and simplifying complex ones at scale.” Aaron Martin, EVP, Chief Digital Officer



















Banner Health brings AI driven virtual assistants for better patient experience

Buoy Health AI triage botTrue to their mission “Making health care easier, so life can be better”, Banner Health has invested in digital health technologies to support their patient experience.
Banner Health, a large regional health care system headquartered in Phoenix, manages 28 acute hospitals across 6 western states, with 5,000+ Banner Health doctors and specialists and over 50 urgent care centers.
Given trends towards consumerism and digital health usage, Banner understands that many patients begin their care journey with internet searches, at a time when these patients feel sick and most vulnerable. Sorting through search responses, patients have the burden of finding trusted health information, interpreting what they need and figuring out where to go to address their latest health issue. Unfortunately the default care setting for patients in the United States is the emergency room, which results in $32 billion in annual avoidable ED visits.
Banner Health Triage Bot
When Banner’s Digital Business group looked for potential solutions, they wanted to understand the patient’s journey and explore digital solutions to complement their high touch hospital experience.
“Through research, we learned that our patients often do not know what is causing their symptoms or what to do about it. They do not know whether to go see their primary care doctor, visit an urgent care facility or go directly to the hospital ER”, explains Dr. Jeff Johnson, Banner Health’s Vice President of Innovation and Digital Business.
With their commitment to delivering both a superior patient experience and reduce unnecessary spending, Banner has tested the idea of empowering customers with a digital tool containing clinical content to engage and triage them to the right care setting.
During the fall of 2019, Banner Health evaluated different digital triage tools and selected Buoy Health’s AI assisted chat bot after both an internal clinical leadership evaluation and patient testing for value and desirability.
As part of their solution evaluation, Banner recruited patients who had gone to a doctor, ER or Urgent Care in the last 30 days and asked them to think about the symptoms that had driven their visit while testing several digital triaging tools.
“Our patients commented that the Buoy bot was ‘easy to use’ and they liked not having to log in to use it. Patients felt the bot interaction was also ‘easy to understand’ and was ‘credible’. These patients did not ‘second guess’ the information that they received in the bot exchange”, shares Johnson.
While preparing for the launch, Banner worked closely with their clinical team, informing them about the triage tool and how it works. If the patient mentions using the bot, Banner wanted clinicians to be aware of the digital tool, acknowledge that the patient had invested time into learning about her condition before arriving and recognize that it is part of the Banner patient experience.
In early January, Banner soft launched the Buoy Health bot across all 28-hospitals with a “Get Care Now” link on their website, giving customers the option of interacting with this tool or calling their Nurse on Call.
After clicking on the “Get Health Now” button, patients are asked questions about their profile (e.g. gender, age) and specific symptoms (e.g. which ones, how long, better/same/worse than before). Banner likes that the bot gets smarter with each patient interaction. Since the Buoy digital tool is available to consumers on the internet and patients across different health care systems, this bot has over 3 million interactions per month.
To evaluate this digital tool, Banner gathers feedback from patients about their bot interaction. Specifically the “Buoy bot askes the patient at the end of an exchange to rate the experience and share what she plans to do next (e.g. wait and see, doctor, urgent care, ER). So far we have received excellent star ratings for the tool, 4.7 out of 5”, adds Johnson.
“Working closely with Buoy, we have learned about the patient’s expectations for their experience with this type of digital tool. The bot exchange cannot be too short so that the patient doesn’t trust the responses. And the text exchange cannot be too long so that the patient feels the tool doesn’t know what it is doing”, Johnson comments.
Banner’s Digital Business team is looking closely at bot tool usage and feedback to guide further development. The bot reporting will give them insight into which care setting patients were directed, what questions they asked and which symptoms and conditions were discussed in the exchange with the patient.
In the first 3 weeks after launch, the Buoy triage bot has had 1,200+ users. Moving beyond the soft launch, Banner plans to incorporate the triage bot in their mobile app and to promote it as an option when patients call the Ask the Nurse line. Banner is discussing developing a Triage Summary Report which captures the information that the patient has already provided to share with her doctor. Banner is also looking to expand care settings to include directing the patient to a virtual visit in the right situation.
Banner Health Emergency Department (ED) Bot
Lifelink ED chabot
Like many of the ER experiences across the country, Banner Health patients want more information and communication about what is happening and what will be happening next to adjust their expectations.
The Banner Health team brainstormed about how to help patients get the information that they need throughout their visit without having to trying to find their doctor, walk up to the nurses station or wait for the nurse to come to their room.
Johnson shares that “we found an interesting chatbot solution from a start- up, Lifelink. We started slowly by having patients use the bot on their smart phones while in the ED and we manually responded to their questions. This gave us tremendous insight into what patients wanted to know and when. Based on this insight, we designed the chatbot (available both in English and Spanish) to give the patient a status from the time her labs (or images) are ordered to the time they are ready and then reviewed by her doctor. We needed to set expectations that the lab will take 45 mins before it is done. In order to enable these real time updates on labs and images, we integrated the LifeLink bot into our EMR. We promote that the patient signs up to our patient portal to access all of the notes, labs and images from her ED visit”.
Since the ED bot was rolled out across all 28 Banner Hospitals, there are over 100,000 patient users, one million plus conversations with an average of 5-10 conversations with the bot per person.
“We have seen our patient satisfaction score for our EDs increase by up to 35%. This ED bot is such a satisfier”, Johnson explains. “Here are some patient comments about the value of using our ED bot”:
“It was nice to be given updates instead of just sitting in a chair and waiting for someone to get to me.”
“It gave me updates on my progress without bothering the nurses.”
“Very informative, relieved my husband’s worries by 90%”
In the near future, Banner Health is planning to “connect the dots” and engage the patient from the moment of interest and throughout their care journey. “We are looking to begin at the first part of the journey when the patient goes online to let us know that they are coming to the ED. Our ED bot will let the patient know that ‘we will be waiting for her’, provide map, picture of the entrance and parking directions. We also want to have the ED bot support the patient post discharge, reminding her to fill her medication, the date of her follow up visit, signs of what to look for and where to go if there is a relapse”, Johnson explains.
Later in this year, Banner Health is planning to extend the bot to inpatient care, with the goals of helping the patient get the highest outcomes and reducing their length of stay. This inpatient bot project is part of a bigger ‘patient flow’ initiative. Banner wants to get patient’s engaged at the beginning of their stay for a successful discharge. The bot will give guidance and recommendations so that the patient improves her eating, walking and bowel movement activities. Banner is also exploring ways to engage the family/care network to ensure an optimal recovery.













