Modern Testing for the New Modal Aerosol Model

In 2024, the MAM4xx Atmospheric Aerosol library (Modal Aerosol Model – 4 Modes, in C++[xx]) was incorporated into E3SM‘s next-generation atmosphere model, EAMxx. Since then, the MAM4xx team has been working to align development and testing practices with those of the EAMxx developers and greater E3SM project. Rigorous verification and validation testing is a foundational tenet of both projects, following the well-known mantra that “code that is untested, should be untrusted.”

Figure 1. An example of the code coverage test bot to verify test coverage of the codebase.

MAM4xx includes a testing infrastructure with nearly 650 tests that cover approximately 93% of all lines in the codebase (Fig. 1). Running these tests allows the team to verify and validate the correctness of solutions provided by the code. This practice also serves the equally-important purpose of certifying that MAM4xx runs properly and correctly on the current major models of CPU and GPU (Fig. 2). Different generations of hardware can sometimes behave surprisingly differently (especially when it comes to GPUs), and conducting this type of testing manually would be a full-time job for multiple developers. For this reason, most modern code bases employ automated testing, for which there are a variety of strategies.

Figure 2. A summary of the test suite, including both CPU and GPU (CUDA) tests.

One method employed by the MAM4xx team, and also by EAMxx, involves the usage of Sandia National Laboratories’ second-generation Autotester software, known as AT2. AT2 enables GitHub-dispatched continuous-integration (CI) testing to be run on hardware residing on Sandia’s network and premises, and it does so in a manner that strictly adheres to Department of Energy security practices. This opens up ready access to leading-edge computing resources like NVIDIA‘s flagship GPU, the Hopper-architecture H100 graphics card. Testing on Sandia’s systems also allows testing with much shorter queue times than the leadership class machines, enabling more frequent and in-depth testing than the nightly runs on those machines. That means any new code a developer submits for review is automatically tested (Fig. 3) on multiple important hardware platforms before a human reviewer even considers merging the changes into the main-line production code base. In addition, production code is auto-tested on a nightly basis to be sure that no bugs enter the main version of code used by EAMxx and E3SM.

Figure 3. A screenshot of the test suite that runs in every merge request.

Overall, this upgrade to MAM4xx’s test suite gives the developer team more confidence in the codebase and easy access to critical hardware systems.