Evaluating the level of degree programmes in higher education. Design of an empirically underpinned instrument

Abstract

The level of bachelor and master programmes needs to be valid and reliable for taking well-founded decisions on its teaching, learning and assessment. Moreover, the degree programme level needs to be demonstrated validly and reliably in the accreditation system. However, the problem is: What outcomes can validly and reliably represent the degree programme level? Procedure and instrumentation to solve this problem are not yet available. Studies in the UK, the US and by the OECD indicate that there is no appropriate instrument available to evaluate the degree programme level validly and reliably. The evaluations of the Bologna process indicate how difficult it is to prove the degree pro­gramme level validly and reliably. The overall research question is therefore: With what instrumentation can the degree programme level in higher education be evaluated validly and reliably? Four studies have been conducted to answer this question: the generic model of the Educational Level Evaluator (ELE) was designed in the first study and the empirical evidence to evaluate the degree programme level validly and reliably was generated in three other studies. The ELE was conceptually designed with the Design Research Methodology. Four critical factors were developed: the level concept has to be clear and the instrumentation should be transparent, empirically proven, and feasible. The ELE is a procedure to evaluate the degree programme level validly and reliably. This was empirically proven in three empirical studies. In the first empirical study the ELE was applied in one domain of degree programmes in The Netherlands. In the second study the ELE was conducted after having analysed the grades. In the third study the ELE was carried out cross-nationally: degree programmes from five countries participated. In each of the studies 400 – 600 participants were involved. The ELE was conducted in three steps in the empirical studies. 1) The degree programme level was conceptualised in disciplinary thinking and professional attitude, and themes were developed by analysing the external surroundings, current curriculum, and the professional and academic field. This step resulted in themes which had been validated by various stakeholders. 2) The themes were specified in learning outcomes representing the aimed degree programme level. For this aim supporting instruments were used and the learning outcomes were content validated by teachers. 3) These outcomes were modified in a questionnaire for recent graduates. The results refer to the components, measuring the themes, explaining variance and scale reliability to a satisfactory degree.