Researchers at the National Institutes of Health say that artificial intelligence may unknowingly increase racial disparities after finding models were able to accurately identify self-reported race in several different types of radiographic scans, with no explanation as to how.

The scientists designed AI applications that could predict race solely from chest X-rays but found that they also predicted race with high accuracy from mammograms, cervical spine radiographs and CT scans, regardless of the anatomic location.

They also determined race even in images that were highly degraded, cropped to one ninth of their original size or had resolution modified to the point where they barely resembled X-rays.
The researchers were unable to explain how the models did this, indicating that the information they use to make these predictions is not yet known and that AI may unknowingly incorporate racial data, leading to potential biases.

“For AI to truly benefit all patients, we need a better understanding of how these algorithms make their decisions to prevent unintended biases,” said Dr. Judy Gichoya, first study author and the National Institute of Biomedical Imaging and Bioengineering Data and Technology Advancement (DATA) National Service Scholar, in a statement.

Prior research has proven that several potential factors can make AI algorithms racially biased, including data sets not representative of different patient populations, and phenotypes disproportionately present in subgroup populations, such as racial differences in breast or bone density.

Gichoya and her colleagues assessed three large data sets of a diverse patient population with their algorithms. They looked at potential factors that could affect features in radiographic images, including body mass index, breast density, bone density, or disease distribution, but found no answers as to how the algorithm detected race.

“There has been a line of thought that if developers ‘hide’ demographic factors—like race, gender, or socioeconomic status from the AI model, that the resulting algorithm will not be able to discriminate based on such features and will therefore be ‘fair’. This work highlights that this simplistic view is not a viable option for assuring equity in AI and machine learning,” said NIBIB DATA Scholar Rui Sá.

Researchers at Harvard and MIT had similar findings in a study published earlier this year where an AI algorithm predicted race with 90% accuracy just by reading X-ray scans, even when filters were applied.

One theorized that X-ray and CT scanners may detect higher melanin content of darker skin and embed this information in digital images, but says more research is required to validate this.

The NIH study is part of The Medical Imaging Data Resource Center.

The findings were published in Lancet Digital Health.

CT Homepage