Cognitive Laboratory Experiences : On Pre-testing Computerised Questionnaires

Abstract

In the literature on questionnaire design and survey methodology, pre-testing is mentioned as a way to evaluate questionnaires (i.e. investigate whether they work as intended) and control for measurement errors (i.e. assess data quality). As the American Statistical Association puts it (ASA, 1999, p. 11): “The questionnaire designer must understand the need to pretest, pretest, and then pretest some more.” Clark and Schober (1992, p. 29) indicate why this need to pre-test: “Surveyors cannot possibly write perfect questions, self-evident to each respondent, that never need clarification. And because they cannot, the answers will often be surprising.”

In this Ph.D. thesis I have tried to systematically describe my experiences with pre-test research at the Questionnaire Laboratory at Statistics Netherlands, a cognitive laboratory which started its work in 1992. This text is not aimed at a theoretical discussion of cognitive laboratory methods, but focuses on the application of these methods: setting-up and carrying-out pre-test research, analysing the data and presentation of the results.

The thesis starts with an introduction of cognitive laboratory research, including the history of the CASM (Cognitive Aspects of Survey Methodology) movement and the history of the Questionnaire Laboratory at Statistics Netherlands. The next two chapters address aspects of computer-assisted interviewing. Since at Statistics Netherlands most social-survey questionnaires are computerised, this sets the conditions for pre-test research at the Questionnaire Laboratory. Chapter 2 discusses computer-assisted interviewing in general; chapter 3 addresses the effects of computer-assisted interviewing on data quality. These three chapters are introductory to the chapters that follow, the actual core of the thesis, in which the application of cognitive laboratory methods, including several case studies, are discussed.

The methods used at the Questionnaire Laboratory at Statistics Netherlands are discussed in chapters 4 and 5. Chapter 4 presents an overview of pre-test methods. Expert appraisal, focus groups, in-depth interviewing (including follow-up probing, meaning-oriented probing, paraphrasing, targeted test questions, and vignettes), and behavioural coding are discussed from a practical point of view, i.e. how they are applied in the Questionnaire Laboratory. Computer-Assisted Qualitative Interviewing (CAQI) is discussed in chapter 5. The CAQI method has been developed at the Questionnaire Laboratory to pre-test computerised questionnaires. With CAQI a pre-test protocol is integrated in a computerised questionnaire to be tested.

The next four chapters present case studies of cognitive research in which the methods addressed in the chapter 4 and 5 have been applied. These chapters discuss the design and the results (i.e. identified problems in the questionnaire and recommendations for improvement) of these studies.

Chapter 10 concludes this thesis with a summary. A number of identified problems in the investigated questionnaires are: vague or unclear wording, complex syntax, long question, double-barrelled question, conflict with previous question(s), question asks for specific information that is not available by heart, difficult to come to an answer because of complex calculation, overlapping or missing response items. In the conclusion, the identified problems are related to design errors in the questionnaire.

References

ASA, 1999, Designing a Questionnaire. ASA series: What is a Survey? (American Statistical Association, Alexandria, VA. www.amstat.org/sections/SRMS)

Clark, H., and M. Schober, 1992, Asking Questions and Influencing Answers. (In: Tanur (ed.), Questions about Questions: Inquiries into the Cognitive Bases of Surveys: pp. 15-48. Russel Sage Foundation, New York.)