Replication of Cohen’s d: Quantifying Evidence using the Bayes Factor

Abstract

This study introduces and demonstrates two new methods to quantify the evidence for replication of Cohen’s d using the Bayes factor. The present study defines successful replication as the degree to which Cohen’s d in the replication study is in line with the Cohen’s d found in the original study. Non-successful replication is formalized in two hypotheses related to relevant and nonzero differences in Cohen’s d respectively between the original and replication study. The Bayes factor that quantifies the relative evidence for successful replication versus relevant differences outperforms the Bayes factor that concerns nonzero differences in Cohen’s d. The main conclusion of this paper is that researchers should include approximately 100 participants per group in both the original and replication study to be able to acquire convincing evidence in favor of or against successful replication, if differences of 0.50 in Cohen’s d between the original and replication data set are deemed relevant. However, the required sample sizes increase up to 600 participants per group if differences in Cohen’s d of 0.20 are considered to be relevant. The accompanying software quantifies the evidence for replication of Cohen’s d for t-tests or post-hoc tests in analysis of variance (ANOVA), as is demonstrated using a data example.