Retrospective batch analysis to evaluate the diagnostic accuracy of a clinically deployed AI algorithm for the detection of acute pulmonary embolism on CTPA

Abstract

PURPOSE: To generate and extend the evidence on the clinical validity of an artificial intelligence (AI) algorithm to detect acute pulmonary embolism (PE) on CT pulmonary angiography (CTPA) of patients suspected of PE and to evaluate the possibility of reducing the risk of missed findings in clinical practice with AI-assisted reporting. METHODS: Consecutive CTPA scan data of 3316 patients referred because of suspected PE between 24-2-2018 and 31-12-2020 were retrospectively analysed by a CE-certified and FDA-approved AI algorithm. The output of the AI was compared with the attending radiologists' report. To define the reference standard, discordant findings were independently evaluated by two readers. In case of disagreement, an experienced cardiothoracic radiologist adjudicated. RESULTS: According to the reference standard, PE was present in 717 patients (21.6%). PE was missed by the AI in 23 patients, while the attending radiologist missed 60 PE. The AI detected 2 false positives and the attending radiologist 9. The sensitivity for the detection of PE by the AI algorithm was significantly higher compared to the radiology report (96.8% vs. 91.6%, p < 0.001). Specificity of the AI was also significantly higher (99.9% vs. 99.7%, p = 0.035). NPV and PPV of the AI were also significantly higher than the radiology report. CONCLUSION: The AI algorithm showed a significantly higher diagnostic accuracy for the detection of PE on CTPA compared to the report of the attending radiologist. This finding indicates that missed positive findings could be prevented with the implementation of AI-assisted reporting in daily clinical practice. CRITICAL RELEVANCE STATEMENT: Missed positive findings on CTPA of patients suspected of pulmonary embolism can be prevented with the implementation of AI-assisted care. KEY POINTS: The AI algorithm showed excellent diagnostic accuracy detecting PE on CTPA. Accuracy of the AI was significantly higher compared to the attending radiologist. Highest diagnostic accuracy can likely be achieved by radiologists supported by AI. Our results indicate that implementation of AI-assisted reporting could reduce the number of missed positive findings.