Facilitating and enabling large-scale, hyper-resolution, groundwater modeling with distributed-memory parallel computing

Abstract

Water managers and policymakers worldwide face the major challenge of securing the availability of fresh groundwater under excessive groundwater extraction and climate change. To this end, they need future projections of groundwater resources computed with numerical groundwater models. These models must have a sufficiently high spatial resolution (i.e., sufficiently small grid cell size) to capture the relevant physical processes. This has raised the call for models with grid cells that are "hyper resolution", i.e., with sizes that are less than or equal to 1 km. However, applying hyper-resolution numerical groundwater models at larger scales typically results in long runtimes and large memory requirements. To solve this problem, this research investigated the potential of distributed-memory parallel computing. MODFLOW, the world’s most widely used groundwater simulation code, was fully parallelized including the linear Krylov solvers applying the additive Schwarz preconditioner. Experiments were conducted on the Dutch national computer cluster, up to a (relatively small) maximum of 1024 processor cores. Two real-world existing groundwater models were facilitated, that still use structured grids, as well as two new applications were enabled that use more flexible (quad-based) unstructured grids. For facilitating existing models, the (quantitative) integrated National Hydrological Model of the Netherlands (NHM) was considered and the Sand Engine model, a (qualitative) 3D variable-density groundwater flow and salt transport model. Orthogonal recursive bisection partitioning was applied, and strong parallel scaling was evaluated. Large, obtained speedups with a relatively low number of cores (speedup of 22 and 86 with 64 and 256 cores, respectively), show that the applied parallelization could significantly increase the practical applicability of these existing groundwater models. As a first new enabled application, GLOBGM was developed, the world's first time-dependent global groundwater model with a resolution of 30 arc seconds (~ 1 km at the Equator). The METIS graph partitioner was applied in both a straightforward and (hydrological) area-based manner. Three continental-scale groundwater models and one for the remaining islands were derived, for a total of 278 million grid cells. Necessarily, parallel pre-processing of input data was applied. With a relatively low number of cores (382 cores in total), 58 years could be computed in parallel in one night (corresponding to a speedup of 138 with 224 cores for the largest Afro-Eurasia model). This demonstrated that GLOBGM could also be used by modelers who lack access to very large computer clusters. As a second new enabled application, a multi-resolution groundwater model was explored for the Netherlands to incorporate regional-scale models in the NHM. Again, area-based METIS partitioning was applied, but now for a smaller set with a larger variation in size. For evaluating weak parallel scaling, nationwide grid refinements up to a regional-scale resolution of 12.5 m were considered, resulting in a model with more than one billion grid cells. With a relatively low number of cores (up to 469 with a speedup of 326), 8 years could be computed in parallel in 2 days. This shows that these very large groundwater models are already within reach with today's computers.