MOAB-Based Coupler Achieves Bit-for-Bit Parity with Legacy System

The coupler is an important infrastructure component of E3SM, so-called because it couples the different components of the model. That is, it manages the exchange of fluxes and state variables among the atmosphere, ocean, sea ice, land, and river model components. It is responsible for mapping fields between different grids, enforcing conservation, and orchestrating the timing of data transfers across components running concurrently on large parallel systems. Building on the legacy Model Coupling Toolkit (MCT)-based coupler, the project is implementing a new Mesh Oriented DatABase (MOAB)-based coupler, aiming to provide identical scientific results while enabling more scalable, flexible, and modern coupling capabilities for current and next-generation high performance computing platforms.

This new coupler offers improvements in multiple areas:

• More flexible input handling: MOAB allows input data to be stored once, at a single resolution, and mapping weights to be computed online. This enables regridding to any chosen model grid without regenerating input datasets each time.
• Better performance and scalability: MOAB avoids data structures that grow with the number of MPI tasks. Its memory footprint scales only with the local problem size, which is important for high resolution simulations and large task counts.
• Support for dynamic meshes and coastlines: With a full mesh description and online calculation of mapping weights, MOAB can support dynamically changing meshes and land/sea boundaries during a simulation. This capability is important for future experiments such as sea level change studies.
• Alignment with E3SM’s C++ direction: MOAB is a modern C++ code base that aligns with E3SM’s broader transition to C++. It also simplifies coupler design by tracking state internally and enables geometry based load balancing via Zoltan.

In its development, this new coupler has made impressive progress, achieving results identical to the older MCT-based coupler, a key milestone in ensuring reliability. The MOAB coupler has been developed as a “Frankencoupler”: a combination of the MCT and MOAB coupler source code. It started as a copy of the MCT coupler and MOAB datatypes, and interface functions were added incrementally. While initial developments focused on a goal of acceptable roundoff differences that occur during the online map generation workflows and using it for projections, the team decided to go for bit-for-bit identical results with MCT as a validation step. The MOAB coupler can read exactly the same mapping weight files as MCT (and read them in parallel) and compute the projections using SparseMatrix Vector applications (SpMV) similar to MCT. Thus there was no reason the MOAB coupler could not produce bit-for-bit identical results. This was achieved after fixing the order in which the local contributions were accumulated and by adjusting the overly aggressive optimization flags that were used by default in MOAB. Once the values were matching in internal calculations between MCT and MOAB data types, the team switched to ensuring that the coupler history files match bit-for-bit as well, using the cprnc tool. History files now match across all major simulation cases, including fully coupled systems and special cases like atmosphere-only and ocean-sea-ice-only setups. The MOAB driver passed 109 tests in the integration suite on its first try, demonstrating its reliability and readiness for broader use.

The MOAB coupler was successfully tested on advanced leadership computing platforms like Perlmutter, Aurora, and Polaris, ensuring it can handle the demands of modern high-performance systems. After updating the flux and fraction calculations to use MOAB data, there are no longer any dependencies on MCT in the “Frankencoupler.” This means the MCT parts can now be removed to expose the MOAB-only coupler and it can be turned “on” as the default coupler.

Compared to its predecessor, the MOAB coupler introduces a more efficient and flexible design. It uses a fully parallel, low-memory approach to represent surface meshes, making it faster and more scalable. The meshes are described completely using the full connectivity information, and at the same time, can utilize point cloud representations for efficiency as needed (in some data models). It supports online remapping during initialization (exposed as a namelist option) and offers flexible algorithms for repartitioning the underlying computational mesh using geometric or graph-based schemes. Beyond coupling, the MOAB library is being used for many preprocessing tasks like generating remapping data offline and processing topography information for high-resolution cases, showcasing its versatility. Looking ahead, the team plans to remove the remaining MCT components from the “Frankencoupler”, profile and optimize performance, and enable GPU support, ensuring the MOAB coupler remains a cutting-edge tool for advancing Earth system modeling.