Estimating Survey Questionnaire Profiles for Measurement Error Risk

Abstract

Surveys differ in their topics, language, style, and design, and consequently, in their sensitivity to measurement error. Survey literature presents a range of characteristics of survey items that are assumed to be related to the magnitude and frequency of measurement error. In terms of questionnaire design and testing, it would be very useful to have a questionnaire profile that is a summary of the characteristics of the items contained in a questionnaire. This holds especially true in the context of multi-mode surveys where the detection of measurement error is crucial. The questionnaire profiles may be derived from scores that coders assign to the items in a questionnaire. Given that agreement among coders may be relatively low, as we observe, the number of coders must be large to ensure sufficient precision of the profiles. For multiple surveys, the coding workload may then become infeasible. In this paper, we propose methodology for the estimation of questionnaire profiles when a pool of coders is randomly allocated to a series of surveys. The methodology is based on multiple imputation and applied to 11 general purpose surveys in the Netherlands.