Rating and ranking preparedness characteristics important for veterinary workplace clinical training: a novel application of pairwise comparisons and the Elo algorithm
Routh, Jennifer; Paramasivam, Sharmini Julita; Cockcroft, Peter; Wood, Sarah; Remnant, John; Westermann, Cornélie; Reid, Alison; Pawson, Patricia; Warman, Sheena; Nadarajah, Vishna Devi; Jeevaratnam, Kamalan
(2023) Frontiers in Medicine, volume 10
(Article)
Abstract
Quantitatively eliciting perspectives about a large number of similar entities (such as a list of competences) is a challenge for researchers in health professions education (HPE). Traditional survey methods may include using Likert items. However, a Likert item approach that generates absolute ratings of the entities may suffer from the
... read more
"ceiling effect," as ratings cluster at one end of the scale. This impacts on researchers' ability to detect differences in ratings between the entities themselves and between respondent groups. This paper describes the use of pairwise comparison (this or that?) questions and a novel application of the Elo algorithm to generate relative ratings and rankings of a large number of entities, on a unidimensional scale. A study assessing the relative importance of 91 student "preparedness characteristics" for veterinary workplace clinical training (WCT) is presented as an example of this method in action. The Elo algorithm uses pairwise comparison responses to generate an importance rating for each preparedness characteristic on a scale from zero to one. This is continuous data with measurement variability which, by definition, spans an entire spectrum and is not susceptible to the ceiling effect. The output should allow for the detection of differences in perspectives between groups of survey respondents (such as students and workplace supervisors) which Likert ratings may be insensitive to. Additional advantages of the pairwise comparisons are their low susceptibility to systematic bias and measurement error, they can be quicker and arguably more engaging to complete than Likert items, and they should carry a low cognitive load for respondents. Methods for evaluating the validity and reliability of this survey design are also described. This paper presents a method that holds great potential for a diverse range of applications in HPE research. In the pursuit quantifying perspectives on survey items which are measured on a relative basis and a unidimensional scale (e.g., importance, priority, probability), this method is likely to be a valuable option.
show less
Download/Full Text
Keywords: survey, questionnaire, Likert, comparison, rating, ranking, preparedness, methods
ISSN: 2296-858X
Publisher: Frontiers Media S.A.
Note: Funding Information: The authors would like to thank the eleven University of Surrey academic staff and veterinary surgeons who took part in the pilot to test an early version of the survey. The authors would like to thank the local champions at the participating veterinary schools who distributed the main survey locally. The authors would like to thank all 901 participants who took part in the main survey. The authors would like to thank Christian Gilson for his technical support. This paper has been deposited in the preprint server EdArXiv (https://doi.org/10.35542/osf.io/e34r2). Funding Information: JRo is funded by the Longhurst Legacy at the University of Surrey. Publisher Copyright: Copyright © 2023 Routh, Paramasivam, Cockcroft, Wood, Remnant, Westermann, Reid, Pawson, Warman, Nadarajah and Jeevaratnam.
(Peer reviewed)
See more statistics about this item