Automatic coding of dialogue acts in collaboration protocols

Abstract

Although protocol analysis can be an important tool for researchers to investigate the process of collaboration and communication, the use of this method of analysis can be time consuming. Hence, an automatic coding procedure for coding dialogue acts was developed. This procedure helps to determine the communicative function of messages in online discussions by recognizing discourse markers and cue phrases in the utterances. Five main communicative functions are distinguished: argumentative, responsive, informative, elicitative, and imperative. A total of 29 different dialogue acts are specified and recognized automatically in collaboration protocols. The reliability of the automatic coding procedure was determined by comparing automatically coded dialogue acts to hand-coded dialogue acts by a human rater. The validity of the automatic coding procedure was examined using three different types of analyses. First, an examination of group differences was used (dialogue acts used by female versus male students). Ideally, the coding procedure should be able to distinguish between groups who are likely to communicate differently. Second, to examine the validity of the automatic coding procedure through examination of experimental intervention, the results of the automatic coding procedure of students, with access to a tool that visualizes the degree of participation of each student, were compared to students who did not have access to this tool. Finally, the validity of the automatic coding procedure of dialogue acts was examined using correlation analyses. Results of the automatic coding procedure of dialogue acts of utterances (form) were related to results of a manual coding procedure of the collaborative activities to which the utterances refer (content). The analyses presented in this paper indicate promising results concerning the reliability and validity of the automatic coding procedure for dialogue acts. However, limitations of the procedure were also found and discussed.