Informed Design of Mixed-Mode Surveys: Evaluating mode effects on measurement and selection error

Abstract

“Mixed-mode designs” are innovative types of surveys which combine more than one mode of administration in the same project, such as surveys administered partly on the web (online), on paper, by telephone, or face-to-face. Mixed-mode designs have become increasingly popular in international survey research, because they can increase response and coverage of inexpensive single-mode data collection designs, such as web or paper. This dissertation explains and illustrates methodology for making informed choices for modes used in mixed-mode designs. After a general introduction to the topic of mixed-mode research, each substantial chapter addressed one of three design objectives suggesting different methodological approaches, which are illustrated for the case of the Dutch Crime Victimization Survey (CVS). For this purpose a large-scale mixed-mode experiment was conducted in collaboration Statistics Netherlands in 2011, in which four modes of administration were administered in a split-ballot between-subject design with a follow-up survey (re-interview) in a single-mode of administration (face-to-face). The first design objective concerns the extent of selection error evoked by single-mode survey designs and candidate mixed-mode designs that are considered for implementation in the field. In increasing response and coverage rates of single-mode designs, mixed-mode designs should reduce the selection error of estimates against true population scores, hence increasing representativeness of survey modes with lower response rates. In chapter two, it is demonstrated how the re-interview data as well as frame information can be used to estimate selection error and compare its size across designs. The second design objective concerns the impact survey modes have on the accuracy of survey measurements. Mode effects (or ‘measurement effects’) can bias the estimates of mixed-mode surveys, if one or more of the modes in the design-mix measures certain questions with larger error. In designing mixed-mode surveys it is therefore relevant to evaluate the impact of modes on the size of systematic and random measurement error. A problem in comparing measurement error across survey modes is its entanglement with selection error in field experiments and mixed-mode surveys, also called confounding of measurement and selection effects. Chapters three and four provide methods to estimate differences random and systematic measurement from split-ballot experimental data. The third design objective, finally, concerns the trade-off of selection and measurement error in the total survey error of single- and mixed-mode designs. For example, mixed-mode surveys may decrease selection error, but such decrease may be outweighed by an increase in measurement error. In the absence of true scores to evaluate these errors, it is demonstrated in chapter five how to use single-mode benchmark data (e.g., face-to-face) as substitutes to true scores. Empirical findings suggest that measurement effects were the pre-dominant source of total error difference between survey modes. Mode differences in selection error were generally negligible and the impact of the face-to-face survey on the selection error of the single-mode surveys was very small. The majority of differences lay between the interviewer-administered and self-administered modes. It is discussed that mixtures of interview and non-interviewer modes may lead to increased measurement error of the CVS design.