The E3SM Nonhydrostatic Dynamical Core
Turbulent eddies in water vapor at 500 mb from a 3 km horizontal resolution SCREAM simulation of baroclinic instability using simple physics.
Phase two of the E3SM project includes several next generation development (NGD) efforts aimed at creating new science and computational capabilities for E3SM versions 3 and 4. One of the NGDs is the NGD Nonhydrostatic Atmosphere project led by Peter Caldwell that is focused on developing a Simple Cloud-Resolving E3SM Atmosphere Model (called SCREAM), which will run at a target resolution of 3 km globally. A major benefit of such high resolution is that it explicitly resolves large deep-convective events, which are critical for vertical transport of heat, moisture, and momentum in the tropical troposphere. In lower resolutions deep convection has to be parameterized which proves difficult because it strongly depends on the interplay between small-scale processes such as condensational heating, turbulent mixing, and cloud microphysics. Avoiding uncertainties due to deep convective parameterization is particularly attractive because differences in climate sensitivity between climate models have recently been linked to their convective parameterizations (e.g. Sherwood et al., Nature 2014).
SCREAM will be running on the Oak Ridge National Laboratories’ Summit supercomputer, which contains 27,000 NVIDIA v100 GPUs.
In order to reach its performance target of 1 simulated year per day on upcoming exascale machines, this project is targeting a simplified set of parameterizations (including P3 microphysics (Morrison and Milbrandt, JAS 2015), SHOC macrophysics/turbulence/shallow convection (Bogenschutz and Krueger, JAMES 2013), and RTE-RRTMGP radiation) and is being rewritten from scratch in C++ using the Kokkos library for performance portability on both GPU and CPU machines. A major feature of the NGD Nonhydrostatic Atmosphere project is the use of a rigorous development process where all code is tested for correctness, convergence and performance. The decision to switch to C++ follows the successful implementation of a C++/Kokkos version of the E3SM hydrostatic dynamical core (Bertagna et al., 2019) funded by the DOE CMDV Software Modernization project (PI: Andy Salinger).
Reflection of sunlight by clouds is critical to climate, but isn’t captured by conventional General Circulation Models (GCMs). SCREAM aims to explicitly resolve deep convection (the smooth-textured anvil-shaped cloud in the picture) while parameterizing the smaller cumuli.
At 3 km resolution, nonhydrostatic effects become important. For that reason, SCREAM is using a nonhydrostatic version of the E3SM dynamical core, called HOMME-NH, developed under the recently completed DOE SciDAC project, “A Nonhydrostatic Variable Resolution Atmosphere Model in E3SM” (PI: Mark Taylor). HOMME-NH allows for run-time switching between hydrostatic (for low resolution) and nonhydrostatic formulations. Much of the discretization is identical to E3SM’s original hydrostatic dynamical core. Nonhydrostatic enhancements include a reformulation of the thermodynamic equation in terms of potential temperature, additional prognostic variables for geopotential height and vertical velocity, and a horizontally-explicit-vertically-implicit (IMEX) time-stepping scheme. Tracer transport will be handled by a new conservative semi-Lagrangian approach developed under the DOE SciDAC project, “Non-Hydrostatic Dynamics with Discontinuous Galerkin Methods” (PI: Pete Bosler).
The E3SM team extensively tested HOMME-NH using idealized problems developed for the Dynamical Core Model Intercomparison Project (DCMIP 2012 and 2016). Two of these test cases – the splitting supercell thunderstorm (Zarzycki et al., 2019) and the moist baroclinic instability with simplified physics (Reed & Jablonowski, 2012) – prepared the project for SCREAM’s ultra-high resolutions. These test cases verified the code’s scalability (including initialization and input/output) and tested the team’s regridding software for initial data set preparation and model output post-processing, thereby demonstrating the project’s computational readiness for extremely high resolutions. Researchers ran the thunderstorm test case for 2 simulation hours at 1 km global resolution, producing 4.1 TB of data. The baroclinic instability test ran for 40 simulation days at 3 km resolution, using 131,000 cores of NERSC’s Cori-KNL computer.
Specific humidity (water vapor concentration) at an elevation of ~500 mb in the Earth’s atmosphere. Results are from a moist baroclinic-instability idealized test case running at 3 km horizontal resolution (average grid spacing at the Equator) for 40 days. The instability is triggered by adding a small perturbation in the northern hemisphere to a geostrophically-balanced initial state. The counter at the top shows the time in days and hours. The animation starts at day 4 since very little happens before then. The instability grows exponentially to day 9, eventually spreading to the southern hemisphere, with the flow becoming fully turbulent around day 30. Specific humidity ranges from zero (represented by the darkest purple color) to 7 g/kg (represented by the brightest shade of yellow).
The animation above was produced using high-frequency output of the specific humidity field from the baroclinic instability test case. In this test case, a small perturbation to a geostrophically-balanced initial state triggers baroclinic instability which grows into chaotic multiscale structures. This instability is responsible for much of the eddy activity (weather) in Earth’s midlatitudes. Next steps include testing HOMME-NH with more realistic atmospheric simulations, first using the full E3SM suite of atmospheric parameterizations and then transitioning to the SCREAM parameterizations when they become available.
