Reliability and validity of emergency department triage systems

Abstract

Reliability and validity of triage systems is important because this can affect patient safety. In this thesis, these aspects of two emergency department (ED) triage systems were studied as well as methodological aspects in these types of studies. The consistency, reproducibility, and criterion validity of the Manchester Triage System (MTS) were studied in chapter 2. The consistency and reproducibility appeared to be ‘substantial’ and ‘high’. However, substantial percentages mistriage occurred. This was further studied in chapter 3 by reviewing weighting schemes for calculating a weighted kappa. It was found that the existing weighting schemes rewarded mistriage too extensively. Therefore, triage weighted kappa was developed and applied. Triage weighted kappa decreased the reliability of triage systems substantially. Besides these shortcomings, the value of kappa depends on the distribution of the data. Therefore, in chapter 4 an approach was developed on how to interprete and compare the reliability of triage systems. The reliability should be interpreted by plotting the kappa against the minimum, normal, and maximum kappa, to obtain information about the skewness of the data. Comparisons between systems can be made by calculating a normal kappa a so that equal chance corrections are obtained. Because this approach affects sample size calculations, in chapter 5 sample size calculation methods for reliability studies were reviewed. Currently no sample size calculation methods exist for reliability studies conducted to ordinal measurement scales such as triage systems. When calculating sample sizes it is important to take into account the dependence of kappa on the distribution of data by setting the null- and alternative hypothesis to the maximum kappa or by setting the confidence interval width to a maximum of 0.10. In chapter 6 the construct validity of the MTS and Emergency Severity Index (ESI) were studied. The ESI was more strongly associated with hospital admission than the MTS. Concerning ED mortality, patients in the more urgent categories of both systems were at increased risk of dying. A small number of patients did not require ESI resources but were admitted to the hospital or sent to the outpatient department. In chapter 7 it appeared that these patients were most likely elderly, referred, or presenting with ‘post operative complications, wound care problems and plaster problems’. Because of this small but structural flaw the ESI guidelines needs revisions concerning these patient groups. Chapter 8 focussed on pain assessments at triage with the MTS. Pain was assessed in nearly one third of the patients who required an assessment according to the guidelines. Nurses mentioned several reasons for not assessing pain at triage including overtriage, time constraints and difficulties with interpreting pain. The reported reasons indicated that the MTS guidelines should describe more precisely how to assess pain. Although both systems’ guidelines need revisions, the studies described in this thesis indicate that the ESI is more reliable and valid than the MTS. Methodologically, one should critically review existing weighting schemes before applying one, and account for the distribution of data when interpreting reliability. These suggestions will reflect the reliability more in accordance with clinical practice.