Comparisons of means using exploratory and confirmatory Approaches

Abstract

This article discusses comparisons of means using exploratory and confirmatory approaches. Three methods are discussed: hypothesis testing, model selection based on information criteria, and Bayesian model selection. Throughout the article, an example is used to illustrate and evaluate the two approaches and the three methods. We demonstrate that confirmatory hypothesis testing techniques have more power—that is, have a higher probability of rejecting a false null hypothesis—and confirmatory model selection techniques have a higher probability of choosing the correct or the best hypothesis than their exploratory counterparts. Furthermore, we show that if more than one hypothesis has to be evaluated, model selection has advantages over hypothesis testing. Another, more elaborate example is used to further illustrate confirmatory model selection. The article concludes with recommendations: When a researcher is able to specify reasonable expectations and hypotheses, confirmatory model selection should be used; otherwise, exploratory model selection should be used.