Comparing open data benchmarks: Which metrics and methodologies determine countries' positions in the ranking lists?

Abstract

An understanding of the similar and divergent metrics and methodologies underlying open government data benchmarks can reduce the risks of the potential misinterpretation and misuse of benchmarking outcomes by policymakers, politicians, and researchers. Hence, this study aims to compare the metrics and methodologies used to measure, benchmark, and rank governments' progress in open government data initiatives. Using a critical meta-analysis approach, we compare nine benchmarks with reference to meta-data, meta-methods, and meta-theories. This study finds that both existing open government data benchmarks and academic open data progress models use a great variety of metrics and methodologies, although open data impact is not usually measured. While several benchmarks’ methods have changed over time, and variables measured have been adjusted, we did not identify a similar pattern for academic open data progress models. This study contributes to open data research in three ways: 1) it reveals the strengths and weaknesses of existing open government data benchmarks and academic open data progress models; 2) it reveals that the selected open data benchmarks employ relatively similar measures as the theoretical open data progress models; and 3) it provides an updated overview of the different approaches used to measure open government data initiatives’ progress. Finally, this study offers two practical contributions: 1) it provides the basis for combining the strengths of benchmarks to create more comprehensive approaches for measuring governments’ progress in open data initiatives; and 2) it explains why particular countries are ranked in a certain way. This information is essential for governments and researchers to identify and propose effective measures to improve their open data initiatives.