Exascale Computing Project Develops a Super-parametrization in E3SM

An illustration of the super parameterization concept. In each atmospheric grid cell (left), the vertical column of traditional parameterizations is replaced with a 2D cloud resolving “superparameterization” model (right).

The E3SM project team is working to develop a cloud-resolving earth system model with the throughput necessary for multi-decade, coupled high-resolution climate simulations. This next-generation model has the potential to substantially reduce major systematic errors in precipitation found in current models because of its more realistic and explicit treatment of convective storms. Consequently, it will improve the ability to assess regional impacts of climate change on the water cycle that directly affect multiple sectors of the US and global economies, especially agriculture and energy production.

The E3SM-MMF project is part of the Exascale Computing Project (ECP), is led by Mark Taylor (Sandia National Laboratory) and is integrating a cloud-resolving convective parameterization (super-parameterization) into the DOE Energy Exascale Earth System Model (E3SM) using the Multiscale Modeling Framework (MMF), and will explore its full potential to scientifically and computationally advance climate simulation and prediction.

Convective storm system nears the Chicago metropolitan area
http://www.spc.noaa.gov/misc/AbtDerechos/derechofacts.htm

The impact of climate change on the global and regional water cycle is one of the highest priorities and most difficult challenges in climate change prediction. While changes in the frequency of droughts, floods, and severe storms are anticipated, there is little predictive skill in how they will be manifested, particularly on regional scales. Current earth system models possess limited ability to model the complex interactions between the large-scale, mostly two-dimensional baroclinic atmospheric motions and the smaller scale three-dimensional convective motions found in clouds and individual storms. These motions and their interactions, to first order, determine the spatial distributions and characteristics of regional precipitation. Complexities include the microscale chemistry and physics of cloud formation as well as the impacts of anthropogenic climate change on cloud formation. Properly resolving the key processes involved in cloud formation requires resolution (grid spacing) on the order of 1 km in the atmosphere. It is possible to run such resolution on today’s O(10) petascale (10^15) computing systems, but only at great expense and for very short times (several simulated days). Running conventional climate models at this resolution, for 100-year simulations requires a 5000x increase in computing resources. Furthermore, to accurately address the evolution of regional climate in the presence of large natural variability requires 10-100 member ensembles. Such a simulation campaign would require zettascale (10^21) computing resources.

Supercomputers: Titan and Summit at the Oak Ridge Leadership Class Facility and Aurora at the Argonne Leadership Computing Facility

For exascale (10^18) computing, the E3SM project thus considers a multiscale modeling framework (MMF) approach to cloud-resolving modeling often referred to as superparameterization, which offers significant opportunities for unprecedented model skill improvement that has yet to be fully explored due to limited computing resources.

The E3SM-MMF project will integrate a cloud-resolving convective “super” parameterization into the DOE E3SM earth system model and will explore its full potential to scientifically and computationally advance climate simulation and prediction. The E3SM-MMF superparameterization is being designed to make full use of GPU accelerated systems (including Titan and Summit, at the Oak Ridge Leadership Class Facility) and also involves refactoring and porting other key components of the E3SM model for GPU systems. In FY19  the scope of the project will be extended to also target the Aurora 2021 system.

The overarching challenge is to develop an earth system model with a fully weather-resolving atmosphere and cloud-resolving superparameterization, an eddy resolving ocean and ice components, all while obtaining the necessary throughput to run 10-100 member ensembles of 100-year simulations.

The challenge is formidable, but when developed, the E3SM with MMF will be the first to run climate simulation campaigns with cloud resolving convection – removing one of the largest sources of uncertainty – cloud treatment – in the model. It will substantially reduce major systematic errors in precipitation with realistic and explicit convective storm treatment. It will also improve the ability to assess regional impacts of climate change on the water cycle that directly affect multiple sectors of the U.S. and global economies (agriculture & energy production).

A usual way to characterize the performance of a model is through a Figure of Merit (FOM). The E3SM-MMF’s FOM is defined as the throughput of a cloud- resolving earth system model. For the baseline, a comparison against the standard E3SM model running at a global 3km resolution will be performed.  For the FOM speedup, the baseline FOM will be compared to the performance of the E3SM-MMF model running with a cloud-resolving convective parameterization on Summit, with GPU acceleration. This FOM speedup will combine algorithmic speedup from the MMF approach, and GPU acceleration. The cloud resolving convective parameterization will be run at a resolution of at least 3km and potentially as fine as 1km.

Throughput is measured in simulated-years-per-day (SYPD) without I/O. I/O is excluded to simplify the benchmarking procedures. Those tasks focused on improving the I/O infrastructure measure their performance through I/O rate benchmarks.
To estimate throughput of the full earth system model, standardized benchmarks of simpler configurations are used, such as atmosphere-land only, or ocean-ice only. Benchmarking these components separately makes it much easier to collect strong scaling data.  Based on the performance data collected in the component simulations, a very good estimate of the performance of the coupled system can be obtained via

time_to_solution = 1.2*max( ocean_time, atmosphere + ice time).

This formula is based on current simulation data which shows that coupling between components adds 20% to the overall cost, and component concurrency (that atmosphere and ice model run sequentially w.r.t. each other on the same nodes, while the ocean model runs concurrently on a different set of nodes).

The E3SM-MMF project was first funded by ECP in 2016, and recently went through a very successful review.