AI chatbot responses lack readability, actionability regarding … – Urology Times
SPOTLIGHT –
“Our findings show that AI chatbots provided accurate information with little misinformation. However, the information was provided at a college reading level and with little actionability," says Abdo E. Kabarriti, MD, FACS.
Artificial intelligence (AI) chatbots generally provide accurate information for queries regarding urological cancers, but the information tends to lack in readability, understandability, and actionability, according to findings recently published in European Urology.1
For the study, the investigators assessed the quality of information from 4 AI chatbots: ChatGPT, Perplexity, Chat Sonic, and Microsoft Bing AI.
“This article is the first of its kind to examine how AI chatbots perform with regards to cancer-related inquiries. This is of great importance as more people turn to AI to obtain information,” said senior author Abdo E. Kabarriti, MD, FACS, in correspondence with Urology Times®. Kabarriti is an assistant professor at SUNY Downstate Health Sciences University in Brooklyn, New York.
For the study, the investigators assessed the quality of information from 4 AI chatbots (ChatGPT, Perplexity, Chat Sonic, and Microsoft Bing AI) on prostate, bladder, kidney, and testicular cancer-related queries. The top 5 search queries for each cancer from January 2021 to January 2023 according to Google Trends were used and analyzed for quality, understandability, actionability, misinformation, and readability.
Findings showed moderate to high scores regarding the quality of information, with a median DISCERN score of 4 out of 5 (range, 2 to 5). Responses specifically related to cancer treatment tended to have lower DISCERN scores among all the chatbots, given a lack of information regarding treatment mechanisms, risks, benefits, and their effects on quality of life. Among all 4 chatbots, ChatGPT demonstrated the lowest median DISCERN score of 3.
The responses for all chatbots were not found to contain much misinformation as demonstrated by a median Likert score of 1 out of 5, where a lower score indicates less misinformation. The most cited sources of information for 3 of the 4 chatbots were The Mayo Clinic, Cleveland Clinic, and the American Cancer Society. ChatGPT was the only chatbot that did not cite any sources.
Further, understandability was only found to be moderate, with a median score of 66.7% (range, 44.4% to 90.9%). The information also tended to be presented at a difficult reading level according to Flesch-Kincaid readability levels, with a median score of 11.8 (range, 5.7 to 38.1). Chatbot responses tended to use medical terminology, and responses were often brief (median word count of 100), which the authors noted “may not explain information in a comprehensive manner.”
Actionability was also found to be moderate to poor across all 4 chatbots, with a median PEMAT-P score of 40% (range 0 to 40%). Responses on prostate cancer queries showed the lowest median understandability and actionability scores compared with the other urologic cancers included in the study.
Kabarriti concluded, “Our findings show that AI chatbots provided accurate information with little misinformation. However, the information was provided at a college reading level and with little actionability. AI chatbots will undoubtedly play a big role in the future, and our study suggests that they should be used to complement the patient-clinician discussions.”
Reference
1. Musheyev D, Pan A, Loeb S, Kabarriti AE. How well do artificial intelligence chatbots respond to the top search queries about urological malignancies? Eur Urol. Published online August 9, 2023. Accessed September 1, 2023. doi:10.1016/j.eururo.2023.07.004
Study shows promise of immunotherapy treatment for penile cancer
“These findings provide encouraging evidence that immune checkpoint inhibitors can be effective in treating a subset of patients with penile squamous cell carcinoma," said co-first author Amin Nassar, a clinical fellow at Yale School of Medicine.
Destigmatizing Urology: Dr. Pettaway discusses penile cancer
“When you compound the patient delay and…the physician delay, then you really get a delay in diagnosis,” says Curtis A. Pettaway, MD.
Surgery found to be a feasible stand-alone treatment for early metastatic testicular cancer
"We are confident that surgery for [testicular seminoma] will be included into treatment guidelines in the near future,” says Sia Daneshmand, MD.
Dr. Richard Matulewicz discusses smoking cessation counseling and treatment among urologists
“Because so many disease processes that we deal with are related to tobacco use and smoking specifically… I think this is absolutely our domain,” says Richard S. Matulewicz, MD, of tobacco cessation counseling and treatment.
Northwestern Urology holding free course on Advances in Genitourinary Robotic Reconstruction
The program will feature leaders in GU robotic reconstruction and will emphasize novel techniques and innovation in managing complex reconstructive challenges.
UCI receives renewal as Orange County’s only federally designated comprehensive cancer center
The National Cancer Institute has again recognized the UCI Chao Family Comprehensive Cancer Center as one of the nation’s top cancer centers, renewing its “comprehensive” designation
2 Clarke Drive
Cranbury, NJ 08512
609-716-7777
