October 18, 2024

Language model “UroBot“ surpasses the accuracy of experienced urologists

Language model “UroBot“ surpasses the accuracy of experienced urologists

Scientists at the German Cancer Research Center (DKFZ), together with doctors from the Urological Clinic of the Mannheim University Hospital, have developed and successfully tested a chatbot based on artificial intelligence. “UroBot” was able to answer questions from the urology specialist examination with a high degree of accuracy, surpassing both other language models and the accuracy of experienced urologists. The model justifies its answers in detail based on the guidelines.

With advances in personalized oncology, urological guidelines are becoming increasingly complex. Whether in the tumor board, on the ward or in the practice, a precise second-opinion system for medical decisions in urology could support doctors in evidence-based and personalized care, especially when time or capacity is limited.

Large language models (LLMs) such as GPT-4 have the potential to retrieve medical knowledge and answer complex medical questions without additional training. However, their applicability in clinical practice is often limited due to outdated training data and a lack of explainability. To overcome these hurdles, a team led by Titus Brinker of the DKFZ developed “UroBot”, a specialized chatbot for urology that was supplemented by the current guidelines of the European Society of Urology.

UroBot is based on OpenAI’s most powerful language model, GPT-4o. It uses a customized method of retrieval-augmented generation (RAG) that is able to retrieve relevant information from hundreds of documents in a targeted manner in response to the individual question in order to provide precise and explainable answers. The modified model was tested on 200 specialist questions from the European Board of Urology and evaluated in several rounds.

UroBot-4o answered questions on the specialist examination correctly 88.4 percent of the cases, outperforming the most up-to-date model GPT-4o by 10.8 percentage points. This means that UroBot not only outperforms other language models, but also exceeds the average performance of urologists in the specialist examination, which is reported in the literature as 68.7 percent. In addition, UroBot shows a very high degree of reliability and consistency in its answers.

UroBot’s answers can be verified by clinical experts, since the software identifies the decisive sources and text sections: “The study shows the potential of combining large language models with evidence-based guidelines to improve performance in specialized medical fields. The verifiability and the very high accuracy at the same time make UroBot a promising assistance system for patient care.”The use of comprehensible language models like UroBot will become extremely important in patient care in the next few years and will help to ensure guideline-based care across the board, even as therapy decisions become increasingly complex,” says Brinker.

The research team has published the code and instructions for using UroBot to enable future developments in urology, as well as in other medical fields.

Martin J. Hetz, Nicolas Carl, Sarah Haggenmüller, Christoph Wies, Jakob Nikolas Kather, Maurice Stephan Michel, Frederik Wessels, Titus J. Brinker: Superhuman Performance on Urology Board Questions Using an Explainable Language Model Enhanced with European Association of Urology Guidelines.
ESMO Real World Data and Digital Oncology 2024, DOI: https://doi.org/10.1016/j.esmorw.2024.100078

Original News

Our latest News

discover more
DFG awards Heidelberg research project on early detection of spinal degeneration nearly €250,000 in funding

DFG awards Heidelberg research project on early detection of spinal degeneration nearly €250,000 in funding

The aim of the project is to use special quantitative MRI to visualize early changes in the intervertebral disc and enable timely treatment. Ultrashort echo time (UTE) MRI opens up completely new approaches to preventive imaging that are difficult or impossible to achieve with conventional MRI methods. Full text in German below. DFG fördert Heidelberger […]

Glioblastoma study shows: Molecular diagnostics could enable targeted therapies without delay

Glioblastoma study shows: Molecular diagnostics could enable targeted therapies without delay

Researchers from Heidelberg University’s Medical Faculty, the German Cancer Research Center (DKFZ), and the National Center for Tumor Diseases (NCT) in Heidelberg have tested the use of targeted drugs in the initial treatment of newly diagnosed glioblastoma, an extremely aggressive brain tumor, in the multicenter study “NCT Neuro Master Match (N2M2)”. Of the five drugs […]

Global effort to sequence all life on Earth ramps up

Global effort to sequence all life on Earth ramps up

The Earth BioGenome Project is accelerating its efforts tenfold to sequence all life on Earth by 2035 Summary A global collaboration that includes experts at EMBL’s European Bioinformatics Institute (EMBL-EBI) has entered the second phase of its ambitious plan to sequence all 1.67 million known species on Earth by 2035.  The Earth BioGenome Project (EBP), a […]

GET IN TOUCH

Stay Updated with bioRN’s Newsletter

Sign up for our newsletter to discover more!
* required

BioRN (BioRN Network e.V. and BioRN Cluster Management GmbH) will use the information you provide on this form to be in touch with you and to provide updates and marketing. Please let us know all the ways you would like to hear from us:

You can update your subscription preferences or unsubscribe at any time. Just follow the unsubscribe or update link in the footer of automated emails you receive from us, or by contacting us at info@biorn.org. We will treat your information with respect. For more information about our privacy practices please visit our website: www.biorn.org. By clicking below, you agree that we may process your information in accordance with these terms.

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices.

Intuit Mailchimp