A Semi-Implicit Barotropic Solver for the E3SM Ocean Model
A semi-implicit barotropic mode solver for the E3SM ocean model enables faster and more stable ocean simulations.
Background
General ocean circulation is influenced by motions occurring on a wide range of time scales. Processes that govern the properties of the deep ocean occur on timescales of 600-1000 years, while dynamics near the ocean surface occur on timescales that are orders of magnitude faster. A particular challenge in ocean modeling is the simulation of long timescales while also capturing shorter, climate-relevant timescales. A particular example of this conundrum has been a challenge for ocean modelers for many decades. One of the fastest waves that propagates across the ocean – a gravity wave driven by changes in sea surface height that travels with a vertically uniform motion (known as barotropic) – has a time scale 30-40 times faster than the next fastest type of wave. While it is important to capture some aspects of this barotropic wave mode, it’s representation does not need to be simulated exactly and ocean modelers have taken different approaches to “skip over” this timescale and advance the model 30-40 times faster.
The ocean component of the Energy Exascale Earth System Model (E3SM) – the Model for Prediction Across Scales Ocean (MPAS-O) – solves this barotropic mode by splitting out the two-dimensional equations for this vertically uniform mode from the full three-dimensional ocean and taking many small time steps with this smaller 2D problem, a process known as explicit subcycling. This split-explicit method resolves the fast timescale, but creating a stable solution requires many small (but fast) time steps and extra work to make this solution consistent with the larger three-dimensional solution. An alternative method is to solve a similar system implicitly, by solving a self-consistent matrix equation for the future state using continuously improving estimates of that future state, and converging to this consistent solution iteratively. This implicit method is more stable, but involves many iterations to converge to a solution. Both methods will lead to valid solutions and will permit the rest of the model to advance at the 30x longer time step, but they have different computational characteristics that can lead one to be more efficient than the other. If the implicit method can converge in fewer iterations than the number of small subcycled steps in the explicit method, the implicit method will result in a faster solution. In the past, the implicit method converged slowly and was less efficient, but recent advances in solver technology and system software (global communication operations) have forced a re-examination of implicit schemes.
Science
Scientists working on the “Next Generation Developments in Software and Algorithms” sub-project of E3SM have implemented a semi-implicit barotropic mode solver in the MPAS-O model to enable faster and more stable simulations compared to the existing split-explicit subcycling scheme. When implementing the semi-implicit method, researchers worked to minimize the frequency of global collective processes requiring communication among all the computing nodes, which is one of the most important issues for optimal scalability on modern high performance computing platforms. To this end, the barotropic mode solver uses a preconditioned conjugate gradient (CG) method with the block-Jacobi preconditioner. The preconditioner accelerates the convergence of the iterative solver and thus decreases the number of global collective processes of the CG method by up to 75%.
Figure 1. Strong scaling results of the MPAS-O tested on NERSC’s Cori-KNL machine using a variable mesh from 30 km to 10 km (1.4 million surface cells). Three-dimensional and other computations (black line) are the same for both time-stepping algorithms. SI and SE in a legend refer to the new semi-implicit scheme and existing split-explicit subcycling scheme, respectively.
Impact
This study shows the potential for the semi-implicit time-stepping algorithm to accelerate MPAS-O simulations through several test case experiments across a range of processor cores. The semi-implicit (SI) barotropic mode solver demonstrates better computational efficiency than the existing split-explicit (SE) subcycling scheme without degrading accuracy (see Fig. 1). This new MPAS-O algorithm allows researchers to produce numerical experiments faster using the model in a standalone configuration as well as within the integrated E3SM and with more desirable stability.
Summary
An efficient barotropic mode solver for the MPAS-O has been implemented as a faster and more accurate replacement to the current split-explicit subcycling scheme. The new formulation is obtained by first using a time-step algorithm, known as Crank-Nicolson, that mixes information from both the current time step and estimates of the new solution (thus, semi-implicit). The resulting system has favorable mathematical properties (symmetric and positive definite) that ensure the use of selected iterative methods will converge to a solution. Because computational performance of barotropic solvers is limited mostly by communication of information between processors, a high-performing scheme must both reduce the number of communications per iteration and reduce the total number of iterations. For the first, a special form of the conjugate gradient algorithm is used that packages all communication into a single global sum for each iteration. To reduce the number of solver iterations, the use of a preconditioner can accelerate convergence. In this case a particular form of preconditioner, called block-Jacobi, was constructed from a somewhat simplified (linearized) version of the full barotropic system that also reorders the system for optimal performance. This new preconditioner decreases the iteration count up to 75% compared to a version without preconditioning. The current scheme now demonstrates between 1.4 and two times better computational performance than split-explicit subcycling across a range of processor cores on NERSC’s Cori supercomputer. Initial results from numerical test case experiments that span horizontal resolutions show that the semi-implicit barotropic mode solver produces ocean currents similar to the existing subcycling scheme (see Fig. 2), though more robust testing and evaluation is underway. Next steps include additional numerical improvements such as better preconditioners or solvers, partly through the use of ForTrilinos, a solver library being developed as part of ASCR’s Exascale Computing Project software technologies activities. This work helps to demonstrate the impact of mathematical algorithms on the performance and accuracy of climate models.
Figure 2. Snapshots of kinetic energy field (m2/s2) at the top layer of the ocean at day 100 simulated by the standalone MPAS-O using the new semi-implicit scheme (left) and the existing split-explicit subcycling scheme (right). Real-world initial conditions on a variable horizontal resolution from 30 to 10 km used for this simulation can be found at https://mpas-dev.github.io/ocean/release_6.0/release_6.0.html.
Contact: Hyun-Gyu Kang, Oak Ridge National Laboratory (ORNL)
Funding: This research was supported as part of the E3SM project, funded by the U.S. Department of Energy, Office of Science, Office of Biological and Environmental Research.
