Agarwal, Blueprint: AI and Radiologists, Synthetic Summary

 Regarding the lengthy and heavily mathematical paper AGARWAL, I asked for a 500 word summary (my proxy for a "detailed" summary) using the abstract, intro, and conclusions.

Main blog here:  https://www.discoveriesinhealthpolicy.com/2023/08/journal-club-ai-more-accurate-than.html

Title: Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology

Abstract: The paper investigates the potential benefits and pitfalls of combining human expertise with artificial intelligence (AI) in the field of radiology. While AI has transformative potential, concerns have arisen about its potential to replace human work in complex tasks. However, the paper argues that human radiologists can use AI assistance optimally, and there may be gains from combining human expertise with AI input. The experiment explores how radiologists use AI predictions, whether AI assistance improves their performance, and the optimal form of collaboration between AI and humans. The results show that AI assistance does not consistently improve diagnostic quality, with heterogeneity in treatment effects based on AI certainty. Human radiologists exhibit biases in belief updating, including automation neglect, and treat AI predictions as independent. These findings have implications for designing collaboration between humans and machines.

Introduction: Artificial intelligence is a transformative technology, comparable to the impact of the steam engine and electricity. While AI has shown the potential to outperform humans in predictive tasks, there are arguments that AI should be used as an aid to human radiologists rather than replacing them entirely. The experiment aims to understand how radiologists use AI predictions and how this affects their performance. It investigates potential biases in human decision-making and explores the optimal collaboration between human experts and AI.

Methodology: The experiment involves professional radiologists recruited through teleradiology companies. They diagnose retrospective patient cases, with different information sets manipulated in a two-by-two factorial design. The AI information treatment provides probabilities of potential chest pathology based on an algorithm trained on over 250,000 X-rays. The contextual information treatment provides clinical history information that radiologists typically have but was not used to train the AI. Treatment effects on radiologists' prediction accuracy and decision-making are estimated.

Results: The experiment shows that AI assistance does not consistently improve diagnostic quality on average, despite the AI outperforming most radiologists individually. The lack of improvement is driven by systematic heterogeneity in treatment effects based on AI certainty. AI assistance improves diagnostic quality when the AI prediction is certain but decreases it when uncertain. Radiologists exhibit automation neglect, treating AI predictions independently, and taking more time when AI information is provided.

Conclusions: The study reveals that biases in radiologists' belief updating and sub-optimal use of AI information hinder the potential benefits of AI assistance. An optimal delegation policy suggests that cases should be decided either by the AI alone or the radiologist alone. Only a small share of cases are optimally delegated to radiologists with access to AI. These insights may hold in other settings where experts' main job involves classification tasks. Future research may investigate the benefits of AI-specific training and the influence of organizational and regulatory incentives on human-AI collaboration.

The paper emphasizes the importance of understanding human-AI interaction to shape institutions that guide AI's beneficial use in society. Empirical analysis is particularly valuable in studying human decision-making with AI, given the black-box nature of AI algorithms.