Improving Clouds in E3SM with a Multi-Scale Modeling Framework

Background

There is an ever-growing interest in simulating atmospheric processes at progressively finer resolutions, which comes with a progressively larger computational price tag. The DOE’s recent commitment to invest in GPU-accelerated computers offers the potential for making these expensive scientific experiments a more accessible reality. However, there are limitations to how much GPU hardware acceleration can improve the throughput of a typical atmospheric model. Traditional approaches to atmospheric modeling are ill-suited to efficiently leverage GPU hardware acceleration at finer resolutions due to a concurrent increase in communication demands with increasing computational workload. An alternative approach, known as the multi-scale modeling framework (MMF), or super-parameterization, provides a unique opportunity to reap the benefits of GPU hardware acceleration and expand the range of resolved scales without increasing the amount or frequency of inter-process communication.

The idea behind a multi-scale model is to take two models that are each designed to simulate different scale ranges (i.e., global to synoptic and convective to turbulent) and couple them together across the scale gap between them so that processes at all resolved scales can influence one another (see Fig. 1). This allows the global model to maintain a long time step while still feeling the effects of the short time scale processes explicitly represented by the cloud resolving model (CRM). In this way the MMF approach provides a middle ground between simplified, efficient cloud parameterizations and explicit, expensive global cloud resolving models.

Figure 1. Schematic illustration of the multi-scale modeling framework (MMF). Two models that explicitly resolve two different scale ranges are coupled together such that the smallest scale of the larger model and the largest scale of the smaller model are locked together (algorithmically). In this way the processes across all resolved scales can interact without the two model states drifting apart.

A recent paper by Hannah et al. (2020) documents the result of an MMF configuration for E3SM (E3SM-MMF) in which a CRM was refactored to run entirely on GPUs, significantly increasing the throughput. In addition to hardware acceleration, the model was also modified to use the algorithmic acceleration technique pioneered by Jones et al. (2015), which reduces the number of CRM time steps per general circulation model (GCM) time step without altering the results. The model was further modified to calculate radiative fluxes on groups of CRM columns rather than each column individually, which significantly reduces the substantial cost of calculating radiative fluxes. The combination of these changes increases the throughput of E3SM-MMF from roughly 0.2 to 1.3 simulated years per wall-clock day (sypd), which is more in line with the speed of traditionally parameterized global models. Further acceleration efforts are underway with the goal of increasing the throughput up to 5 sypd.

Model Results

The goal of Hannah et al. (2020) was to compare the results of E3SM-MMF with the standard E3SMv1 and then compare these differences to the results of previous studies with other MMF models. Most of these previous studies focused on comparing the NCAR Community Atmosphere Model (CAM) with it’s super-parameterized (i.e., MMF) counterpart, known as SP-CAM, but there are also results in the literature from MMF versions of the Weather Research and Forecasting model (WRF; Tulich, 2015), the ECMWF Integrated Forecasting System (IFS; Subramanian and Palmer, 2017), the NOAA Climate Forecast System (CFS; Goswami et al., 2015), and the NASA finite-volume GCM (fvGCM; Tao et al., 2009). Generally, the results for E3SM-MMF are similar to these other studies, but the results also reveal a surprising new bias that results from the novel use of an MMF on a spectral element grid.

The pattern of annual mean precipitation in E3SM-MMF is somewhat improved relative to E3SMv1 when compared to TRMM observations from 2000-2004 (see Fig. 2), but there are certain regions where new precipitation biases emerge, such as excessive precipitation over the Northwest Tropical Pacific, around 140 E (south of Japan). E3SM-MMF also shows less precipitation over land, which accentuates pre-existing dry biases in E3SMv1, such as over the Amazon and North American Great Plains.

Figure 2. All‐season 5‐year climatology of total precipitation for TRMM 3B43 (a), E3SMv1 (b), and E3SM-MMF(c). Data are plotted on the native grid in all cases. Units are mm day-1.

Another key finding illustrated in Figure 2 is a grid imprinting bias. This can sometimes appear as a “speckling” overlaid on top of the expected geographic pattern. This bias was initially thought to just be noise in the data, but upon closer inspection scientists realized that it was a systematic bias arising from a curious feedback between the explicitly resolved convection and the spectral element dynamics. The grid of points within each spectral element is structured such that the points along the element edge are treated somewhat differently than the points in the interior of the element, specifically in how gradients are calculated. In areas of active convection the problem manifests as a bias for slightly more precipitation on the element edge points and less precipitation on the element interior points. The nature of this feedback makes it difficult to isolate and quantify, but researchers were able to design a filter that could more or less separate the speckling from the data, and a detailed look at the filtered moisture budget revealed that the effect of the speckling bias was actually quite small. Nonetheless, the unphysical bias has a non-negligible effect, so the team has spent considerable effort on a solution (see below).

One of the areas that E3SM-MMF shows notable improvements relative to E3SMv1 is in the climate variability, especially on diurnal and intraseasonal time scales. Figure 3 shows a lagged regression analysis designed to highlight the strength of the Madden-Julian Oscillation (MJO), which is the dominant mode of sub-seasonal variability in the Tropics and is notoriously absent in most global models. Figure 3 shows 20 to 100 day filtered outgoing longwave radiation (OLR) and 850 mb zonal wind regressed onto an index that roughly represents the MJO made with 20 to 100 day filtered OLR averaged over 80°E to 100°E and 15°S to 15°N. The data were meridionally averaged from 15°S to 15°N to produce a time vs. longitude plot that depicts the typical evolution of the MJO as eastward movement through time, with day 0 representing the peak convective activity over the index region. Compared to observational data from ERA5 and NOAA, E3SMv1 does not produce the correct pattern or amplitude, whereas E3SM-MMF is able to produce a coherent eastward propagating signal closer to what is observed.

Figure 3. Lagged all‐season regression of equatorially averaged, 20‐100 day filtered OLR (shading) and 850‐mb zonal wind (contours) regressed on an index of 20‐100 day filtered OLR averaged over the equatorial Indian Ocean (15°S-15°N, 80–100°E).

Fixing the Grid Imprinting

Figure 4. Illustration of time averaged precipitation from a pair of E3SM-MMF simulations with a ¼ degree dynamics grid and two different physics grids. In the upper left portion the physics columns coincide with the nodes of the spectral element grid, which results in grid imprinting. In the lower right portion of the figure, the physics columns are on a separate, slightly coarser, finite volume grid that resolves the grid imprinting issue. The grid used for dynamics calculations is the same in both cases.

After learning more about the E3SM-MMF grid imprinting bias illustrated in Figure 2, it was obvious that the problem could not be ignored. In E3SMv1 the physics and dynamics calculations are done on the same spectral element (SE) grid, but recent work by Herrington et al. (2019a, b) presented an alternative approach where the physics calculations are done on a separate finite volume (FV) grid. This new approach is often colloquially referred to as “Physgrid”. The SE grid for E3SM uses a 4×4 grid of points within each spectral element, and Herrington et al. showed that similar results can be achieved by using a 2×2 or 3×3 grid of FV cells within each element specifically for the physics calculations. The Physgrid requires data to be mapped back and forth between the dynamics and physics grids, and the way in which this mapping occurs is very important. A new Physgrid mapping method was recently developed for E3SMv2 that is different from the Herrington et al. approach, but has additional benefits, such as allowing for regional mesh refinement (see the New Physgrid and Dycore Methods Speed Up EAM by 2x story).

Another important benefit of the Physgrid is that it breaks the grid imprinting mechanism in E3SM-MMF. Figure 4 shows the time mean precipitation over the Maritime Continent in Southeast Asia for two high-resolution E3SM-MMF cases. One of these (upper left) uses the traditional approach in which the physics columns are on the same SE grid as dynamics, and the other case (bottom right) uses the new approach with physics columns on a coarser FV grid. Figure 4 clearly shows that the new Physgrid makes the grid imprinting disappear, and also yields a much smoother, and more realistic, result.

Future Work

All E3SM-MMF simulations to date have used single moment microphysics, prescribed aerosols, and prescribed ocean surface temperatures, each of which limit the model’s realism. Previous studies have shown that ocean coupling can improve certain aspects of an MMF (Randall et al., 2016), so coupled simulations with E3SM-MMF will be an important area of future work in order to study climate feedbacks and El Niño–Southern Oscillation (ENSO). Future work will also include looking into several nuanced aspects of how the CRM is integrated into future versions of E3SM, which will focus on higher moment microphysics, treatment of turbulent mixing, convective momentum transport, and aerosol interactions.

Related News Article

• New Physgrid and Dycore Methods Speed Up EAM by 2x

References

• Goswami, B. B., Krishna, R. P., Mukhopadhyay, P., Khairoutdinov, M., & Goswami, B. N. (2015). Simulation of the Indian Summer Monsoon in the Superparameterized Climate Forecast System Version 2: Preliminary Results. J. Climate, 28, 8988–9012. https://doi.org/10.1175/JCLI-D-14-00607.1
• Hannah, W. M., Jones, C. R., Hillman, B. R., Norman, M. R., Bader, D. C., Taylor, M. A., et al. (2020). Initial results from the super‐parameterized E3SM. Journal of Advances in Modeling Earth Systems, 12, e2019MS001863. https://doi.org/10.1029/2019MS001863
• Herrington, A. R., Lauritzen, P. H., Taylor, M. A., Goldhaber, S., Eaton, B. E., Bacmeister, J. T., Reed, K. A., & Ullrich, P. A. (2019). Physics–Dynamics Coupling with Element-Based High-Order Galerkin Methods: Quasi-Equal-Area Physics Grid. Mon. Wea. Rev., 147, 69–84. https://doi.org/10.1175/MWR-D-18-0136.1
• Herrington A. R., Lauritzen, P. H., Reed, K. A., Goldhaber, S., & Eaton, B. E. (2019). Exploring a lower‐resolution physics grid in CAM‐SE‐CSLAM. Journal of Advances in Modeling Earth Systems, 11, 1894– 1916. https://doi.org/10.1029/2019MS001684
• Jones, C. R., Bretherton, C. S., & Pritchard, M. S. (2015). Mean‐state acceleration of cloud‐resolving models and large eddy simulations. Journal of Advances in Modeling Earth Systems, 7, 1643– 1660. https://doi.org/10.1002/2015MS000488.
• Randall, D., DeMott, C., Stan, C., Khairoutdinov, M., Benedict, J., McCrary, R., Thayer-Calder, K., & Branson, M. (2016). Simulations of the Tropical General Circulation with a Multiscale Global Model. Meteorological Monographs, 56, 15.1–15.15. https://doi.org/10.1175/AMSMONOGRAPHS-D-15-0016.1
• Subramanian, A. C., & Palmer, T. (2017). Ensemble superparameterization versus stochastic parameterization: A comparison of model uncertainty representation in tropical weather prediction, Journal of Advances in Modeling Earth Systems, 9, 1231– 1250. https://doi.org/10.1002/2016MS000857.
• Tao, W., Chern, J., Atlas, R., Randall, D., Khairoutdinov, M., Li, J., Waliser, D.E., Hou, A., Lin, X., Peters-Lidard, C., Lau, W., Jiang, J., & Simpson, J. (2009). A Multiscale Modeling System: Developments, Applications, and Critical Issues. Bull. Amer. Meteor. Soc., 90, 515–534. https://doi.org/10.1175/2008BAMS2542.1
• Tulich, S. N. (2015). A strategy for representing the effects of convective momentum transport in multiscale models: Evaluation using a new superparameterized version of the Weather Research and Forecast model (SP‐WRF), Journal of Advances in Modeling Earth Systems, 7, 938– 962. https://doi.org/10.1002/2014MS000417.

Funding

• Department of Energy Exascale Computing Project

Contact

• Walter Hannah, Lawrence Livermore National Laboratory