Comparing the Accuracy of Automatic Scoring Solutions for a Text Comprehension Diagramming Intervention

Abstract

Students typically have great difficulty monitoring their comprehension of textual materials. Completing diagrams about causal relations in expository texts has been a successful intervention to enhance the accuracy of students’ reading comprehension judgments (ie, monitoring accuracy), although there is still room for improvement. Such judgments play a role in crucial self-regulated learning decisions that students make such as allocating time and effort, selecting content for restudy, and/or consulting additional sources. The automated scoring of students’ diagram content can provide a basis for strengthening the diagramming intervention with individual and simultaneous feedback to a high number of students. Leveraging an existing human-coded (correct and incorrect) dataset of 6000+ diagram answers (completed in Dutch by 700+ secondary students), we compared different automatic scoring solutions in terms of classification accuracy. Four computational linguistic models for Dutch were identified and tested in combination with four popular machine learning classification algorithms. The best solution reached 81% accuracy (ie, four out of five answers matched the human coding). Depending on the accuracy required for different applications, these results could be used for fully-or semiautomated scorings of students’ answers to generative activities used in reading comprehension interventions.