Conservative Multimoment Tracer Transport

Schematic for semi-Lagrangian tracer transport. The target element with solid blue points is advected backward in time to the deformed element having the solid purple points to compute new values. MPI (Message Passing Interface) communication pattern uses an upwind scheme which requires much less communication than the full halo exchange with all neighboring nodes.

Background

In an atmospheric model, substances carried by the wind are called tracers. Tracers may represent trace atmospheric gasses, chemically reactive species suspended as microscopic solid or liquid particles, pollutants such as soot particles and radioactive fallout, and suspended particulate matter such as volcanic ash and dust. Due to complex nonlinear interactions between tracers and the atmosphere, particularly via clouds, tracer transport is fundamentally important in climate modeling applications. Global climate models, including the E3SM model, require several conditions on their numerical transport scheme to ensure accuracy and preservation of important properties such as total mass conservation.

Tracer transport is part of the global atmosphere dynamics and transport code, called the atmospheric dynamical core. E3SM uses spectral element methods on cubed sphere grids implemented in the architecture-portable and performant High-Order Methods Modeling Environment (HOMME) framework.

A new implementation of atmospheric tracer transport was especially needed because in E3SM version 1 it was the most computationally expensive part of the simulation due to the high number of tracers (e.g., > 40) required to capture the interactions between tracers and clouds, which affect the solar radiation budget, and other climatologically significant chemical processes on land and in the ocean.

Performance of the EAM spectral element dynamical core HOMME on the EAM version 1 high-resolution configuration (1/4-degree, 40 tracers) with Eulerian transport (solid lines) and SL transport (dashed lines). The y-axis shows simulated years per day (SYPD) and the x-axis displays the number of compute nodes. Machines are NERSC Edison (green) and NERSC Cori-KNL (red). The black line indicates the slope of perfect scaling. The new SL method (HOMME/SL) is shown to be over 2x faster on Cori-KNL and over 3x faster on Edison than the previous version of HOMME.

A new conservative transport scheme based on the combination of Semi-Lagrangian (SL) transport algorithms with spectral element spatial discretizations is developed and implemented within the E3SM Atmosphere Model (EAM) version 2. SL time stepping permits large time steps and compact, high order spectral element stencils ensure that a compute node’s required data are available locally. Combining these two strategies significantly reduces communication within a distributed computing environment and improves performance. The long time steps and local computations translate into improved computational efficiency and parallelism. Using a newly-developed family of property preservation algorithms, Communication-Efficient Density Reconstructors, the new transport scheme preserves several properties for each tracer, including mass conservation, which is especially important for climate models due to the long time scales of the simulations. Thanks to the inherent numerical stability of SL methods, stabilizing techniques such as hyperviscosity are no longer required and, consequently, the new transport scheme is capable of resolving finer scale features than the previous version. Additionally, most of the algorithm is independent of the number of tracers, which makes it less expensive to perform simulations that require many tracer species. The new algorithm is designed to map well to parallel distributed next-generation computing platforms and results in a 3-5 times speedup, depending on computer architecture, over the previous transport scheme without sacrificing any of its key traits such as conservation, accuracy, and shape preservation.

The Semi-Lagrangian transport algorithm features:

• Multitracer efficiency: most of the algorithm is independent of the number of tracers
• Parallelism: algorithm design exposes fine-grained parallelism suitable for advanced architectures
• Globally mass conserving, mass tracer consistent and shape preserving
• Implementation is 2^nd order accurate, but higher orders are possible
• Fast: achieves a 2-3x speedup in the full dynamical core compared to EAM v1 with 40 tracers, including at the challenging strong scaling limit (1 mesh element per computing core).

References

• Bosler, P.A., Bradley, A.M., & Taylor, M.A. (2019). Conservative multimoment transport along characteristics for discontinuous Galerkin methods. SIAM Journal of Scientific Computing, 41(4), B870—B902. http://dx.doi.org/10.1137/18m1165943
• Bradley, A.M., Bosler, P.A., Guba, O., & Taylor, M.A. , Barnett, G.A. (2019). Communication-efficient property preservation in tracer transport. SIAM Journal Scientific Computing, 41(3), C161–C193. https://doi.org/10.1137/18M1165414
• Bradley, A.M., Guba, O., Bosler, P.A., & Taylor, M.A., COMPOSE: Compact multi-moment performance-portable semi-Lagrangian methods, 10.5281/zenodo.2552888 (GitHub repository)

Contact

Peter A. Bosler
Sandia National Laboratories

Funding

• Programs
  • Advanced Scientific Computing Research
  • Climate Model Development and Validation
  • Earth System Model Development
  • Scientific Discovery through Advanced Computing
• Project
  • Non-Hydrostatic Dynamics with Multi-Moment Characteristic Discontinuous Galerkin (NH-MMCDG) Methods