Standing on the Shoulders of Giants: Formalizing and Evaluating Prior Knowledge

Abstract

Research makes the greatest progress when it makes use of the results and insights of others. This dissertation explores, proposes, and demonstrates several ways in which information other than the data at hand can be used in an analysis. The first part concentrates in three chapters on acquiring prior knowledge for Bayesian analyses and its impact on the posterior results. First, a procedure to elicit prior information on a correlation from psychologists is developed and evaluated. Second, a simulation study is conducted to demonstrate the impact of prior knowledge in a two-group latent growth model with unbalanced sample sizes. Prior knowledge on the smaller group has the most meaningful impact on the posterior results, especially with respect to the non-null detection rate. Third, a systematic search strategy for prior knowledge is provided and applied in an empirical example. Understandably, prior knowledge on the smaller group in the unbalanced latent growth model was also most difficult to acquire. Experts were very helpful sources in determining the applicability of various empirical studies as prior knowledge. The second part of this dissertation concerns the evaluation of prior information from previous studies: it introduces testing replication by means of the prior predictive p-value. In this manner, researchers can test whether the relevant findings from a new study significantly deviate from what we can expect given the original findings. Relevant findings are captured through an informative hypothesis, which can, for example, concern the sign of relevant parameters, the relative ordering of parameters, or a minimal meaningful (effect size) value. This method is first explained for the specific case of ANOVA studies, where the relative ordering of the groups is often of main interest. Subsequently, it is demonstrated how the prior predictive p-value can also be used to test replication of structural equation models with the Replication R-package. Part II ends with an overview of several methods to evaluate the replication of an ANOVA, and their performance in the context of small samples. The third and final part of this dissertation uses Bayesian Research Synthesis to evaluate the updated evidence from four cohort studies for competing informative hypotheses. The hypothesis with the highest updated posterior model probability is robustly supported by all studies, irrespective of population specifics and measurement methods.