Impact of non-transportable predictor measurements on performance of prediction models: a measurement error perspective

Abstract

Clinical prediction models have an important role in contemporary medicine. A vital aspect to consider is whether a prediction model is transportable to individuals that were not part of the set in which the prediction model was derived. Transportability of prediction models can be hampered when predictors are measured differently at derivation and (external) validation of the model. This may occur, for instance, when predictors are measured using different protocols or when tests are produced by different manufacturers. Although such heterogeneity in predictor measurement across derivation and validation samples is very common, the impact on the performance of prediction models at external validation is not well-studied. Using analytical and simulation approaches, we examined the external performance of prediction models under different scenarios of heterogeneous predictor measurement. These scenarios were defined and clarified using an established taxonomy of measurement error models. The results of our simulations indicate that predictor measurement heterogeneity induces miscalibration of prediction models and affects discrimination and accuracy, to extents that predictions in new observations may no longer be clinically useful. Our work identifies predictor measurement heterogeneity as an important source of unanticipated performance of a prediction model at external validation. Taking the measurement error perspective can be helpful in identifying and predicting effects of heterogeneous predictor measurement across different settings.