Prediction of Study Success – Creation of Magic Zones

Abstract

Of all who enrol at university programmes, some will be successful students, some will not, depending how success is defined. Common definitions of success are a specific Grade Point Average, a specific number of course credits, graduating in time, and of course, other definitions are possible. Universities apply selection procedures to make admission decisions. Incorrect admission decisions are inevitable if more than one, but not all applicants are admitted: admission of students who will fail and rejection of applicants who would have become successful. Selection was approached as a signal detection problem to investigate the predictive validity of admission instruments on both group and the individual level. It also provided information on the effects of the application of specific cut-off scores on the selection outcomes. Signal detection theory describes both sensitivity and specificity of admission instruments and allows prescription of specific instruments and cut-off scores given specified admission goals. Data of three cohorts of Psychology students at Utrecht University were used to retrospectively make hypothetical selection decisions, based on admission instrument scores that were available at the time of application: a secondary school grade average and a psychosocial score, both taken from the application form, and an admission test grade, i.e. the unweighted average of a multiple-choice exam grade and an essay grade from the programme sample applicants attended before enrolment. The signal detection analyses were complemented by qualitative data on the definition and realisation of academic success and a systematic and integrative review on psychosocial factors. We found that a programme sample can be as accurate as secondary school grades in predicting study success. The predictive validity of the psychosocial score was lower. In line with the literature, accuracy of the prediction of study success was far from perfect for all investigated combinations of admission instruments. Moreover, signal detection analyses showed that different admission tools uniquely select different applicants that will become successful students. The predictive validity of the admission instrument scores is stable across time as students progress through the programme, but study success in later stages of the programme is more accurately predicted by first year grades than by admission instrument scores. Furthermore, study success in theoretical courses was predicted with higher accuracy than study success in research skills courses and professional skills courses. Students, teachers and researchers in focus groups agreed that academic success is broader than the knowledge, research skills and professional skills as reflected by study grades. Psychosocial factors such as independence and motivation were deemed major determinants of academic success. The realisation of academic success was agreed to be a community effort. It requires interaction with peers and teachers, development of self-regulatory skills, space for individuality, and supportive relations. The implications of this research: Use signal detection theory to monitor selection and advice. Continue exploring meaningful ways to answers questions on the (im)possibilities of predicting study success. Design programme samples to optimize self-selection of students. Allow teachers and students space to co-create learning experiences.