Constructing Behaviour Profiles for Answer Behaviour Across Surveys.

Abstract

Answers to survey questions may deviate from the truth. We name suboptimal or non-substantive answers ‘undesirable answer behaviour’. The extent to which responses deviate from the ‘true’ value is referred to as ‘measurement error’. The purpose of this thesis was to detect respondent characteristics that are structurally related to measurement error. First, we investigated a large selection of answer behaviours across ten population surveys from CentERdata and Statistics Netherlands on their occurrences and variances by means of multilevel models. Based on the outcomes, we chose to switch to a method that summarizes and visualizes answer behaviour for each survey and behaviour in detail: The construction of ‘answer behaviour profiles’.

Second, we defined the answer behaviour profile as a density distribution that summarizes answer behaviour for a group of respondents (the ‘respondent profile’) or a group of items (the ‘item profile’). By means of simulations, we examined the statistical properties of the behaviour profiles. We showed that a respondent profile becomes more informative and precise as each respondent fills out more items. And we showed that a respondent profile becomes more stable and certain as the group consists of a larger number of respondents. When a respondent profile is both precise and certain, the respondent profile is accurate. The very same applies to item profiles, with the concepts of ‘respondent’ and ‘item’ interchanged. We used real survey data to illustrate the properties of respondent profiles, item profiles, and smoothing in practice.

Third, we examined the potentially consistent relation between respondent characteristics and undesirable answer behaviour across ten population surveys. We constructed respondent profiles for subgroups of respondents that are based on the categories of several respondent characteristics. These characteristics were investigated on their relation to eight specific undesirable answer behaviours. The profile for each characteristics’ category was compared to the profile for all other categories of that characteristic taken together. These comparisons were executed by an adaptation of the robust effect size measure Cliff’s Delta. Although various categories of various characteristics appeared to be related to various behaviours, the occurrence and direction of these relations were substantially dependent on the specific characteristic, survey, and behaviour. This means that a subgroup of respondents frequently showed more of some behaviour than the other subgroups for one survey, while less of that behaviour than the other subgroups for another survey. Thus, we did not find consistent relations between any respondent characteristic and any answer behaviour.

We consider it likely that item characteristics have a more substantial and consistent influence on answer behaviour than respondent characteristics. Therefore, we recommend the use of item profiles to investigate this influence to detect item characteristics that may lead to measurement error. Hence, we anticipated on examining the association between item characteristics and undesirable answer behaviour in follow-up research. We investigated to what degree item characteristics could be determined reliably. The results showed that intercoder reliability was low for various item characteristics. We tried to explain low intercoder reliability and made several suggestions to cope with low intercoder reliability.