WHO's New AI-powered Chatbot SARAH is Giving Wrong Medical Answers

Recently, WHO warned on its website that this early prototype, introduced on April 2, provides responses that “may not always be accurate.”

In a surprising turn of events, the World Health Organization’s (WHO) newly launched AI-powered chatbot ‘SARAH’ has been reported to provide incorrect medical information. The bot, designed to offer instant health advice, has been under scrutiny for its erroneous responses.

SARAH, short for Smart AI Resource Assistant for Health, is a part of the WHO’s campaign to find technology that can both educate people and fill staffing gaps in the world facing a healthcare worker shortage.

Recently, WHO warned on its website that this early prototype, introduced on April 2, provides responses that “may not always be accurate.”

Some of SARAH’s AI training is years behind the latest data. Further, the bot occasionally provides bizarre answers, known as hallucinations in AI models, that can spread misinformation about public health, the WHO revealed.

More on Sarah

WHO unveiled its latest innovation: SARAH, an AI-powered digital health promoter ahead of World Health Day, themed 'My Health, My Right'.

It is a virtual health worker that is available to talk 24/7 in eight different languages to explain topics such as mental health, tobacco use, and healthy eating.

Equipped with enhanced empathetic response capabilities driven by generative AI, SARAH seeks to transform the way people across the globe access health information.

Why SARAH Giving Inappropriate Answers?

The AI-powered chatbot was trained on OpenAI's ChatGPT 3.5, which used data through September 2021. Consequently, it lacks the most recent information on medical advisories and news events.

For instance, when asked whether the US FDA has approved the Alzheimer's drug Lecanemab, SARAH, the chatbot, incorrectly stated that the drug was still undergoing clinical trials. In reality, the drug had received approval for early disease treatment in January 2023.

Further, when asked about the trend in hepatitis deaths, SARAH was unable to provide immediate details from a recent WHO report. It was only after a second prompt to check the agency’s website for updated statistics that SARAH would provide the information. On it, the WHO said it's currently investigating whether there is a delay in updates.

There are also instances where SARAH is unable to provide any response. For example, when a user named Javan asked SARAH for locations to get a mammogram in Washington, DC, the chatbot was unable to provide an answer.

This highlights the limitations of AI chatbots such as SARAH in providing accurate and reliable information in critical areas such as healthcare.

WHO’s Response on this Malfunctioning

According to WHO, SARAH doesn't have a diagnostic feature such as WebMD or Google. It mentioned that the bot is programmed to not talk about anything outside of the WHO's purview, including questions on specific drugs.

SARAH often sends people to a WHO website or says that users should "consult with your health-care provider."

On the matter, Ramin Javan, radiologist & researcher, George Washington University, said, "It lacks depth, but I think it's because they just don't want to overstep their boundaries and this is just the first step."

SARAH is a continuation of a 2021 project by the WHO named Florence. Florence was a virtual health worker that provided fundamental information on COVID-19 and tobacco. The avatars for both Florence and SARAH were developed by Soul Machines, a company based in New Zealand.

Soul Machines does not have access to the data used by SARAH. However, Soul’s CEO, Greg Cross, stated that they are utilizing the GPT data to enhance the results and user experience provided by the chatbot.

However, the use of open-source data such as GPT’s presents risks, as it is often targeted by cybercriminals, warns Jingquan Li, a public health and IT researcher at Hofstra University.

Users accessing SARAH via Wi-Fi could be susceptible to malware attacks or video camera hacking. Despite these potential threats, Jaimie Guerra, communications director, WHO maintains that data breaches should not be an issue due to the anonymity of the sessions.

"These technologies are not at the point where they are substitutes for interacting with a professional or getting medical advice from an actual trained clinician or health provider," Alain Labrique, the director of digital health and innovation at WHO, stated.

Additionally, WHO envisions SARAH working alongside researchers and governments to deliver accurate public health information. The agency is seeking advice on enhancing the bot and its application in emergency health situations while acknowledging that SARAH is still a work in progress.

Earlier this year, the WHO issued ethical guidelines for health-related AI models to its government partners. These guidelines emphasized the importance of data transparency and safety protection.

Is it Safe to Rely on AI Tools for Health Queries?

The topic is an ongoing debate. ChatGPT, a widely used AI chatbot, was recently scrutinized by researchers at Vanderbilt University Medical Center. They presented the chatbot with 284 medical questions, and while it correctly answered most of them, it was found to be “spectacularly and surprisingly wrong” on several occasions. These inaccuracies have raised concerns about the dependability and precision of AI chatbots when it comes to providing medical advice.

Moreover, AI technologies like Deep Neural Networks (DNNs), despite their improved precision and efficiency, are prone to serious errors. These mistakes, which can include recommending inappropriate drugs or overlooking tumor markers on scans, can lead to significant harm and affect a large number of patients.

This highlights the need for rigorous validation and testing of AI systems before they can be entrusted with critical tasks such as dispensing health-related information.