Cognitive representations in raters' assessment of teacher portfolios

Abstract

Portfolios are frequently used to assess teachers' competences. In portfolio assessment, the issue of rater reliability is a notorious problem. To improve the quality of assessments insight into raters' judgment processes is crucial. Using a mixed quantitative and qualitative approach we studied cognitive processes underlying raters' judgments and the reliability of these judgments. Six raters systematically assessed 18 portfolios. The interrater reliability of 12 portfolios was satisfactory. Variance analysis showed slight rater effects. We used the Correspondent Inference Theory (Jones & Davis, 1965) and the Associative Systems Theory (Carlston, 1992; 1994) to analyse judgment forms and retrospective verbal protocols. Raters' cognitive representations on the dimensions abstract-concrete and positive-negative were significantly related to the judgments given and to the reliability of these judgments.