Refactoring the Elastic-Viscous-Plastic (EVP) Sea Ice Dynamics Solver
Figure 1. Single-core performance improves more than 5x compared to the standard
implementation, and up to 35x on high-bandwidth memory hardware.
Refactoring the elastic–viscous–plastic sea ice dynamics solver improves performance on CPU and GPU computer architectures.
The Science
This study reveals two dominant bottlenecks in the computational performance of the elastic–viscous–plastic (EVP) sea ice dynamics solver and effectively addresses them. Rewriting the code (refactoring) to improve memory access patterns overcomes issues with synchronization across processing units, and hardware bandwidth enhancements solve performance scaling issues on single compute nodes.
The Impact
Sea ice dynamics calculations are the most computationally expensive aspect of the sea ice component of earth system models. The refactorization described in this work allows the entire EVP calculation to be performed on a single compute node and has the potential to significantly improve the efficiency of long-term and multiple-ensemble-member earth system simulations.
Summary
The EVP dynamical solver within the sea ice model, CICE v6.5.1, suffers from two dominant computational bottlenecks, namely (1) the number of synchronization points required for parallel computations during each time step combined with the irregular domain of active sea ice points, and (2) the lack of single-instruction, multiple-data (SIMD) code generation.
The standard EVP solver is refactored based on two generic patterns. The first pattern exposes how general finite differences on masked multi-dimensional arrays can be expressed in order to produce significantly better code generation by changing the memory access pattern from random access to direct access. The second pattern takes an alternative approach to handle static grid properties.
The measured single-core performance improvement is more than a factor of 5 compared to the standard implementation, scaling strongly until the available memory bandwidth is saturated (Fig. 1). On architectures with sufficient bandwidth, strong scaling results in a single-node improvement factor of 35 over the standard implementation. This study also demonstrates improved performance on GPU processors.
Although CICE is not the sea-ice component of E3SM, the EVP implementation in E3SM closely follows that of CICE, and therefore these performance improvements could be transferrable.
The efficiency improvements come with little impact in results (Fig. 2).
Figure 2. (a) Sea ice thickness on the Northern Hemisphere (left) and on the Southern Hemisphere (right). (b) Difference between CICE using the standard EVP solver and the refactored EVP solver implemented into CICE on the Northern Hemisphere (left) and on the Southern Hemisphere (right). All results represent 1 January 2009 after 5 years of simulation starting on 1 January 2005 and using the gx1 grid provided by the CICE Consortium.
Publication
- Rasmussen, Till Andreas Soya, Jacob Poulsen, Mads Hvid Ribergaard, Ruchira Sasanka, Anthony P. Craig, Elizabeth C. Hunke, and Stefan Rethmeier. 2024. “Refactoring The Elastic–Viscous–Plastic Solver From The Sea Ice Model Cice V6.5.1 For Improved Performance”. Geoscientific Model Development 17 (17). Copernicus GmbH: 6529-6544. doi:10.5194/gmd-17-6529-2024.
Funding
- This work was supported by the Earth System Model Development program area of the Department of Energy, Office of Science, Biological and Environmental Research program.
Contact
- Elizabeth Hunke, Los Alamos National Laboratory
This article is a part of the E3SM “Floating Points” Newsletter, to read the full Newsletter check:
