Researchers have built a tool to uncover hidden issues in the massive datasets used to train medical artificial intelligence.
The tool searches training data for subtle patterns that could lead AI models to incorrect conclusions, potentially jeopardizing patient care. The work aims to help researchers and regulators make AI more reliable and trustworthy for real-world clinical use.
“The models that drive precision medicine learn to infer clinical outcomes from the data they’re trained on. In many cases that works great, but it can also lead to interesting failures that aren’t immediately apparent,” says senior author Mathias Unberath, a John C. Malone Associate Professor of Computer Science at the Johns Hopkins University. “The tool we developed gives you a clear understanding of which elements of metadata are posing the most risks for your model to pick up a bias.”
The work, a collaboration between Johns Hopkins and the U.S. Food and Drug Administration, is published in npj Digital Medicine.
The “Clever Hans Phenomenon” refers to a horse named Hans that in the early 1900s got famous for being able to do math, tapping out correct answers with a hoof. Turns out, Hans couldn’t do math but was quite good at reading his human’s body language that inadvertently revealed how he should respond. Medical AI models can be a lot like Hans, Unberath says, reaching conclusions by reading the wrong signals.
“Models take the same shortcuts,” he says, pointing to AI trained to accurately predict a person’s sex by looking at their eyes—turns out the model was relying on the presence of mascara rather than any biological features.
“We train models to be the most predictive, but we have zero control over what the model uses to make the prediction.”
The team’s tool detects the potential for such flawed associations.
While others have built safeguards based on how an AI model performs, this new tool, called Generalized Attribute Utility and Detectability-Induced bias Testing, or G-AUDIT, examines the training data itself for signs that a model could learn from unintended cues rather than clinically meaningful signals.
“The most exciting thing is how this is a shift in how we do the work,” says co-author Mitchell Pavlak, a PhD student in Unberath’s lab. “We’re not checking our work after the fact; we analyze the data to figure out what is likely to be a problem.”
G-AUDIT identifies and ranks attributes in the data likely to lead AI to make potentially incorrect conclusions.
Researchers tested it on a range of medical data including images, text, and spreadsheets. It revealed hidden flaws in data collection, imaging conditions, and other elements likely to lead to flawed conclusions about a patient’s health.
For instance, a skin cancer dataset included images from a cancer clinic that served a high-risk population, and images from a general dermatology clinic that saw fewer cancer cases. Because the imaging systems the clinics had varied in quality, an AI trained on the dataset could wrongly deduce that camera quality is a key predictor of cancer—or even that the presence of rulers in images predicts cancer because the cancer clinic tended to include rulers in images to track the size of growths over time.
“Envision taking this algorithm into the real world with a smart phone camera,” Unberath says. “There’s no ruler. The camera quality is completely irrelevant. But the model learned to associate ‘ruler’ and ‘camera.’ You have just created a health disparity.”
The team plans to continue to build out and expand the tool, which they say could be adapted for use beyond healthcare.
“The core takeaway is that right now people are auditing models, but they don’t really know what to audit for. We are changing that,” Unberath says. “If you only audit for things that are top of mind to you, you’re probably missing a lot of things that could lead to spurious correlations.”
Additional authors of this work include Nathan Drenkow, a senior research scientist at Johns Hopkins’ Applied Physics Laboratory; former Johns Hopkins PhD student Keith Harrigian; Ayah Zirikly, an assistant professor at George Washington University and adjunct assistant professor at Johns Hopkins; and Adarsh Subbaswamy, Mohammad Mehdi Farhangi, and Nicholas Petrick of the FDA’s Center for Devices and Radiological Health.
The work was supported by the FDA’s Research Participation Program at the Center for Devices and Radiological Health.