Automatic coding of online collaboration protocols

Abstract

An automatic coding procedure is described to determine the communicative functions of messages in chat discussions. Five main communicative functions are distinguished: argumentative (indicating a line of argumentation or reasoning), responsive (e.g., confirmations, denials, and answers), informative (transfer of information), elicitative (questions or proposals requiring a response), and imperative (commands). A total of 29 different dialogue acts are specified and recognized automatically in the chats of students during collaboration. The validity of the automatic coding procedure was examined using three different types of analyses. First, an examination of group differences was used to provide evidence about the validity of the automatic coding procedure. Ideally, the coding procedure should be able to distinguish between groups who are likely to communicate differently. For example, it has been shown extensively, that women communicate differently than men do: women use more affiliative language, whereas men use more assertive language. The coding procedure was able to mostly replicate these findings. For example, women were found to use more responsive dialogue acts, whereas men used more informative and imperative dialogue acts. Second, to examine the validity of the automatic coding procedure through examination of experimental intervention, the results of the automatic coding procedure of students with access to a tool that visualizes the degree of participation of each student were compared to students that did not have that sort of visualization. It was expected that students with access to the tool would engage in more argumentative interactions. This expectation was partly confirmed as it was found that students with access used more conditional arguments. However, they did not use more reasons, contra arguments, etc. Finally, the validity of the automatic coding procedure was examined using correlation analyses. Results of the automatic coding procedure were correlated with results of a manual coding procedure. This manual coding was aimed at identifying the task-related and social aspects of online collaboration. Because some aspects of the manual coding procedure focused on similar aspects of online collaboration as the automatic coding procedure, moderate to strong correlations were expected. Indeed, several significant correlations were found In conclusion, the results presented in this paper indicate favorable results concerning the validity of the automatic coding procedure for dialogue acts.