Modeling the Climate at Exascale
Key Points
- The new C++/Kokkos E3SM Atmosphere Model (EAMxx) is the first truly exascale global atmosphere model.
- EAMxx (evolved from SCREAM) is ready for science and will be the atmosphere component of the E3SM version 4 release.
- High resolution EAMxx, i.e., SCREAMv1, improves some meso-scale features and the diurnal cycle but large-scale biases require improvement across all four seasons.
In June 2024, a manuscript that documents a fully refactored version of the E3SM atmosphere model capable of running at exascale was published in the Journal for Advances in Earth System Modeling (doi: 10.1029/2024MS004314). This manuscript serves as the main reference for the C++ version of the Simple Cloud-Resolving E3SM Atmosphere Model (SCREAMv1) which was used to run four seasonal simulations at km scale global resolution. SCREAMv1 is part of the evolving E3SM Atmosphere Model (EAM), which will be called EAMxx. EAMxx is set to be the atmosphere component for the release of E3SMv4 and is a significant step towards a fully Exascale Earth System Model. This manuscript was selected for a Research Spotlight on Eos.
Background
A pillar of the U.S. Department of Energy (DOE) Energy Exascale Earth System Model (E3SM) project is to design an Earth System Model capable of leveraging the incredible computational power available to the DOE. In recent years it has become clear that the path to exascale goes through heterogeneous computer systems that employ both CPUs and GPUs. This creates a difficulty for modern legacy software, whose code base had been written at a time when GPUs did not exist. A significant code modernization effort would be required in order to run on CPU/GPU machines. In 2018, the SCREAM project was conceived to do exactly this.
A New Model Designed for the Exascale Future
In the six years since the project started, the SCREAM team has successfully developed the first truly Exascale Global Atmosphere Model capable of running at cloud-resolving (3km) spatial resolution. This accomplishment was recognized by the global community at the Supercomputing meeting in Denver, CO in 2023 with the inaugural Gordon Bell Prize in Climate Modeling. SCREAM is capable of running a global 3.25km simulation with greater than 1 simulated year per day (SYPD) of performance on the DOE Leadership Class machine Frontier (Fig. 1), an unprecedented accomplishment.
Figure 1. EAMxx demonstrates how it is able to maintain strong scaling on mixed CPU/GPU systems, obtaining performance of greater than 1 simulated year per day at a global resolution of 3.25km. (a) shows how Total model (“Model”), atmosphere model (“Atmosphere”), and dynamical core (“Dycore”) scale. The black undecorated line is the perfect-scaling reference. (b) shows how Physics parameterizations (“Physics”) and throughput of the Dycore components scale.
The SCREAM team was able to accomplish this task by completely redesigning and rewriting the atmosphere model infrastructure in C++ using the performance portability library Kokkos. There are many advantages to this approach. The team was able to abandon unused or under-performant legacy software. The new modern infrastructure is written with software best practices in mind, including robust unit testing. The Kokkos library makes transitioning to new computer systems less painful and ensures the model will maintain the same level of performance across architectures.
Such high performance unlocks new avenues of science never before explored in the E3SM community. In this work, the authors were able to run four 40-day simulations spanning all four seasons at a global 3km resolution. This is an extension of a previous study conducted by Caldwell et al. (2021) which was limited to a single season by the computational performance of the original Fortran E3SM atmosphere model. Having a set of simulations that spans the year provided SCREAM researchers with enough data to examine the diurnal cycle and to better understand the benefits and limitations of global km-scale modeling.
The simulations show that despite minimal tuning, SCREAMv1 captures the global top of atmosphere net radiative imbalance to within 1.2 Wm-2 of satellite retrievals across all four seasons. Atmospheric river characteristics and the midlatitude jet are also well represented in these short simulations. The extremely high horizontal resolution also allows for comparisons that have previously been more difficult in a 100-km scale model. For example, convective systems can be produced realistically over the continental US (see Fig. 2).
In addition, simulation output can more readily be compared with Atmospheric Radiation Measurement (ARM) retrievals, because the grid resolution in SCREAM is much closer to the spatial scale at which observations are made. The comparison with ARM retrievals shows that although SCREAMv1 simulates the diurnal development of boundary layer clouds, it also overestimates 10-m wind speeds, overestimates the downward shortwave radiation over ARM sites, underestimates the liquid water path, and underestimates the column-integrated precipitable water path. These findings over the ARM sites also show up in the comparison of SCREAM v1 with global datasets. Other notable model biases include a too dry lower troposphere over much of the tropics, warmer near-surface air temperatures over land, and a compensation in the top-of-atmosphere shortwave radiation where too much absorbed shortwave over the tropics and subtropics is balanced by a too much reflected shortwave over the high latitudes.
Figure 2. The upper panels show maps of precipitation (color) and 500mb geopotential height (lines, which denote the elevation at which the air pressure is 500mb) from IMERG/ERA5 (observations) and SCREAM on April 5. Lower panels show Hovmöller diagrams of precipitation from IMERG and SCREAM, meridionally averaged over the red box in panels a-b. The snapshot of precipitation in the top row shows how SCREAMv1 produces organized convective storms associated with a synoptic wave feature over the central US. The Hovmöller diagram in the bottom right shows how that system propagates eastward, consistent with what we see also from satellite observations (bottom left).
Despite minimal tuning, SCREAM does well at realistically simulating many phenomena. Many of the biases identified in this study are currently under investigation using data from longer simulations, with ongoing analysis of a pair of Cess-style (Cess et al., 1989) year-long simulations, one forced with present-day sea-surface temperatures (SSTs) and one with those SSTs increased uniformly by 4K. Earlier this year the SCREAM project embarked on a campaign to run SCREAM at cloud-resolving scales for a full decade.
Data Availability
Instructions for accessing all simulation data from the SCREAMv1 simulations can be found at 2024 4Seasons SCREAMv1 Paper Simulation Results.
The version of the SCREAM codebase used for all full-length simulations in this manuscript is available via Zenodo at the following DOI: 10.5281/zenodo.10724808
Performance data is available at GitHub – E3SM-Project/perf-data: Performance data store (unstructured vs. structured data in PACE) for E3SM benchmarks, sub-models, and kernels and are archived at Performance data for SCREAM runs for the four-seasons paper. Within this repository, see screamv1-frontier-feb2023/readme.txt and screamv1-summit-oct2022/readme.txt for details.
Reference
- Donahue, A. S., Caldwell, P. M., Bertagna, L., Beydoun, H., Bogenschutz, P. A., Bradley, A. M., et al. (2024). To exascale and beyond—The Simple Cloud-Resolving E3SM Atmosphere Model (SCREAM), a performance portable global atmosphere model for cloud-resolving scales. Journal of Advances in Modeling Earth Systems, 16, e2024MS004314. https://doi.org/10.1029/2024MS004314.
