An interrater reliability analysis of preoperative mortality risk calculators used for elective high-risk noncardiac surgical patients shows poor to moderate reliability

Abstract

BACKGROUND: Multiple preoperative calculators are available online to predict preoperative mortality risk for noncardiac surgical patients. However, it is currently unknown how these risk calculators perform across different raters. The current study investigated the interrater reliability of three preoperative mortality risk calculators in an elective high-risk noncardiac surgical patient population to evaluate if these calculators can be safely used for identification of high-risk noncardiac surgical patients for a preoperative multidisciplinary team discussion. METHODS: Five anesthesiologists assessed the preoperative mortality risk of 34 high-risk patients using the preoperative score to calculate postoperative mortality risks (POSPOM), the American College of Surgeons surgical risk calculator (SRC), and the surgical outcome risk tool (SORT). In total, 170 calculations per calculator were gathered. RESULTS: Interrater reliability was poor for SORT (ICC (C.I. 95%) = 0.46 (0.30-0.63)) and moderate for SRC (ICC = 0.65 (0.51-0.78)) and POSPOM (ICC = 0.63 (0.49-0.77). The absolute range of calculated mortality risk was 0.2-72% for POSPOM, 0-36% for SRC, and 0.4-17% for SORT. The coefficient of variation increased in higher risk classes for POSPOM and SORT. The extended Bland-Altman limits of agreement suggested that all raters contributed to the variation in calculated risks. CONCLUSION: The current results indicate that the preoperative risk calculators POSPOM, SRC, and SORT exhibit poor to moderate interrater reliability. These calculators are not sufficiently accurate for clinical identification and preoperative counseling of high-risk surgical patients. Clinicians should be trained in using mortality risk calculators. Also, clinicians should be cautious when using predicted mortality estimates from these calculators to identify high-risk noncardiac surgical patients for elective surgery.