Quality of epidemiological studies: Procedural rules for uncertain science for policy, a case study on bisphenol-A

Abstract

Abstract This paper proposes a method for in-depth mapping of heterogeneity in expert judgment, in the evaluation of the quality of epidemiological studies used in regulatory chemical risk assessment. Whereas consensus in scientific advisory groups provides legitimation for subsequent political action, it can also have unintended effects on the quality of regulatory risk assessment. Based on empirical testing of our method, called Qualichem_epi, with ten experts and two epidemiological case studies about bisphenol A (BPA)’s effects on human health, we have shown that expert judgment plays an essential role in managing uncertainty and deciding what “quality” of a study actually means. We found substantial heterogeneity of scientists’ judgments about the quality of epidemiological studies, even if the same criteria were used for the assessment. This heterogeneity is not present anymore in reports produced by expert groups, where results are presented under the collective signature of all the scientists involved. We argue that flattening heterogeneity can be an important problem when it is not the result of true scientific agreement but only a secondary effect of consensus-based working procedures of agencies that experts have to follow. Qualichem_epi provides an easy to understand color-based picture of both majority and minority opinions in a scientific advisory group. We suggest that it could be used on a regular basis for communicating quality assessments of epidemiological studies in regulatory chemical risk assessment.