OAK BROOK, Ill. — Researchers evaluating the performance of ChatGPT-4 Vision found that the model performed well on text-based radiology exam questions but struggled to answer image-related questions accurately. The study’s results were published today in Radiology, a journal of the Radiological Society of North America (RSNA).
Chat GPT-4 Vision is the first version of the large language model that can interpret both text and images.
“ChatGPT-4 has shown promise for assisting radiologists in tasks such as simplifying patient-facing radiology reports and identifying the appropriate protocol for imaging exams,” said Chad Klochko, M.D., musculoskeletal radiologist and artificial intelligence (AI) researcher at Henry Ford Health in Detroit, Michigan. “With image processing capabilities, GPT-4 Vision allows for new potential applications in radiology.”
Ad Statistics
Times Displayed: 48844
Times Visited: 680 Reveal Mobi Pro integrates the Reveal 35C detector with SpectralDR technology into a modern mobile X-ray solution. Mobi Pro allows for simultaneous acquisition of conventional & dual-energy images with a single exposure. Contact us for a demo at no cost.
For the study, Dr. Klochko’s research team used retired questions from the American College of Radiology’s Diagnostic Radiology In-Training Examinations, a series of tests used to benchmark the progress of radiology residents. After excluding duplicates, the researchers used 377 questions across 13 domains, including 195 questions that were text-only and 182 that contained an image.
GPT-4 Vision answered 246 of the 377 questions correctly, achieving an overall score of 65.3%. The model correctly answered 81.5% (159) of the 195 text-only queries and 47.8% (87) of the 182 questions with images.
“The 81.5% accuracy for text-only questions mirrors the performance of the model’s predecessor,” he said. “This consistency on text-based questions may suggest that the model has a degree of textual understanding in radiology.”
Genitourinary radiology was the only subspecialty for which GPT-4 Vision performed better on questions with images (67%, or 10 of 15) than text-only questions (57%, or 4 of 7). The model performed better on text-only questions in all other subspecialties.
The model performed best on image-based questions in the chest and genitourinary subspecialties, correctly answering 69% and 67% of the image-containing questions, respectively. The model performed lowest on image-containing questions in the nuclear medicine domain, correctly answering only 2 of 10 questions.
The study also evaluated the impact of various prompts on the performance of GPT-4 Vision.
Original: You are taking a radiology board exam. Images of the questions will be uploaded. Choose the correct answer for each question.
Basic: Choose the single best answer in the following retired radiology board exam question.
