Summary of E3SM’s Modeling Challenges
Schematic of E3SM Model Components
Recently released E3SMv2.0 included many improved features scientifically and computationally as summarized in the 2021 Updated E3SM Phase 2 Overview Brochure. As some E3SM staff begin the v2 simulation campaigns and prepare for phase 3, other team members tackle remaining challenges. Many of these remaining challenges – like splitting of the intertropical convergence zone (ITCZ) across the equator, excessive heat and dry bias in the central US, and anemic rain over the Amazon – are found in most climate models worldwide. This article highlights some unique major challenges of E3SM and their current status. It also describes the Simple Cloud-Resolving E3SM Atmosphere Model (SCREAM) – a new E3SM model configuration aimed at very fine resolution. The third goal of this article is to describe the level of software support the team can provide for DOE/EESSD funded researchers who are interested in using SCREAM for their research projects.
Scientists are encouraged to review the recent Funding Opportunity Announcements (FOAs) focused on Earth System Model Development and Analysis to address the challenges mentioned in this article.
In addition, interested scientists are strongly encouraged to get familiar with and use the E3SM Diagnostics and Analysis capabilities included below, which are built for supporting model analysis across all of the E3SM computation platforms, and connecting new analysis to this system is the most efficient way to integrate new capabilities into the E3SM development cycle.
Atmosphere
Low-level clouds and higher resolution
Photo by Matthew Bowden from FreeImages
Low-level clouds cover vast portions of the earth and act as a net cooling by reflecting incoming solar radiation. Stratocumulus clouds over the eastern subtropical oceans have been notoriously difficult to simulate accurately in Earth System Models (ESMs). Cloud-related air motions occur on a horizontal scale of kilometers or less, and clouds are topped by sharp inversions occurring over tens of meters or less in the vertical. Because ESMs cannot resolve these scales, they rely on approximate parameterizations instead. Improving the representation of clouds in ESMs will require higher resolution (horizontal and vertical) as well as improved cloud parameterizations. Increasing vertical resolution in E3SMv1 in conjunction with an improved cloud parameterization (CLUBB) had only a moderate impact on stratocumulus clouds. Increasing horizontal resolution (from 100 km to 25 km) did not significantly improve stratocumulus clouds either. It is possibly because these increases may not have been sufficient. Recent simulations with the Simple Cloud-Resolving E3SM Atmosphere Model (SCREAM) at 3.25 km resolution showed significant promise in improving stratocumulus clouds (Caldwell et al., 2021).
Aerosol-cloud interactions
Emissions of anthropogenic aerosols (in particular sulfate particles) impact clouds by increasing the number of cloud droplets, leading to brighter and probably longer-lived clouds (i.e. aerosol indirect effects and cloud adjustment). Physically, aerosol-cloud interactions occur at small scales that cannot be resolved by present-day Earth System Models (ESMs) and therefore must be approximated by uncertain parameterizations. Aerosol-cloud interactions globally act as a net cooling term, whose magnitude is poorly constrained. In E3SM (v1 and v2), like other ESMs, the magnitude of this cooling term is excessive based on observational evidence. This results in a lack of warming during the second half of the 20th century when anthropogenic aerosol emissions peak (Golaz et al, 2019). The aerosol-related radiative effects are concentrated in the northern hemisphere where anthropogenic emissions dominate.
Quasi-Biennial Oscillation (QBO)
The quasi-biennial oscillation (QBO) of the zonal mean wind is a prominent feature of the tropical lower stratosphere, and has been shown to influence stratospheric and tropospheric variability. Although recognized for its importance, the QBO has been a challenge to represent in global circulation and Earth system models. In order to obtain a QBO, models need to represent larger scale drivers of the QBO such as the Kelvin and mixed-Rossby gravity waves, generate or parameterize small-scale gravity waves, and have adequate vertical resolution, ∼500 – 700 m in the free troposphere and lower stratosphere, to capture the wave-mean flow interactions. The significantly increased vertical resolution (72 layers) and model top (~ 0.1 hPa) in E3SMv1 makes it capable of adequately representing the main features of the stratospheric circulation. However, the QBO in the default version of E3SMv1 is still too fast and too strong. Although an improved simulation of QBO can be achieved through modifying some convective GW parameterization parameters (Richter et al., 2019), the result would be sensitive to potential changes in convective parameterization in future versions of E3SM. Improvements in the variability of the convective parameterization in E3SM are needed to drive a more robust and realistic QBO.
The Simple Cloud-Resolving E3SM Atmosphere Model (SCREAM)
In addition to the standard E3SM atmosphere model described above, E3SM has developed a next-generation atmosphere model written in C++ and targeted towards global cloud-resolving simulations on exascale computers. A prototype of this model using the conventional E3SM infrastructure and programming model is described in Caldwell et al (2021; JAMES). This prototype F90 version is no longer supported as we finalize and deploy the C++ version. The team anticipates that within 6 months, the new SCREAM v1 code and run scripts for 2 different simulation configurations will be available for OLCF’s Summit and NERSC’s Perlmutter machines. OLCF Frontier and ALCF Aurora configurations will come later. One configuration is set to begin a simulation on Aug 1st, 2016 and the other initializes on Jan 20, 2020. Both cases are taken from the DYnamics of the Atmospheric general circulation Modeled On Non-hydrostatic Domains (DYAMOND) intercomparison and use prescribed sea surface temperatures, sea ice extent, and aerosol forcing.
Idealized Configuration: SCM, RRM and DP-SCREAM
In addition to global simulations, several idealized modes are available. These modes are often much more computationally efficient, are good for zooming in on individual physical processes, and/or are better for comparing against observations at a particular time and place. For example, the single-column model (SCM) exercises E3SM physics for a single column of the atmosphere with horizontal dynamics and boundary conditions specified from observations. Simulations can be performed for more than 20 case studies using pre-defined run scripts available from the E3SM Single Column Model library: E3SM Single Column Model Case Library · E3SM-Project/scmlib Wiki. For SCREAM, the developers extended the SCM framework to enable doubly-periodic cloud-resolving versions of each SCM case study. This limited-area doubly-periodic version of SCREAM (which we call DP-SCREAM) is critical for model development because global runs are too expensive to perform very often. DP-SCREAM won’t be publicly available until a manuscript describing and testing it is submitted (which should happen by March 2022). At that point, the SCM case library described above will be expanded to also include run scripts for DP-SCREAM versions of each case. One limitation to DP-SCREAM is that it assumes the surface is homogeneous. This precludes using it for studying orographic effects or making realistic regional weather forecasts. For those uses, E3SM can be configured as a regionally-refined model (RRM). In RRM mode, the model uses fine grid spacing over the region of interest and coarse resolution elsewhere. Because the number of grid columns in the coarse region is usually negligible compared to the number of columns in the fine-resolution region, RRM computational expense is on par with the cost of a regional model (like WRF), which is why E3SM has no plans to support a regional configuration. One issue with RRM mode is that parameterization settings for high resolution are suboptimal for coarse resolution and E3SM lacks the ability to use different tunings in different regions. The ability to nudge the model state toward reanalysis or output from a different simulation is useful for overcoming this lack. Nudging can be performed in the coarse-resolution region of an RRM while leaving the fine-resolution region untouched. Creating the grid, initial conditions, and boundary conditions for a new RRM configuration is necessarily a complicated process. The steps are described at Running E3SM on New Grids.
Ocean
Atlantic Meridional Overturning Circulation (AMOC)
The Atlantic Meridional Overturning Circulation (AMOC), defined as the zonally integrated component of surface and deep currents in the Atlantic Ocean, is a critical component of the global climate system as it carries and redistributes heat, carbon and nutrient along its path. An accurate representation of the AMOC in earth system models is essential for its baseline climate state and faithful simulations and predictability. Despite the significant progress during past decade in modeling AMOC in the climate science community such as the improved AMOC in eddy resolving high resolution models, challenges remain largely due to its complex nature, its interactions with various processes over multiple scales. In both E3SMv1 and E3SMv2, AMOC is on the order of 10 Sv, considerably weaker than the observed value (the mean AMOC 2004 to 2009 was 18.5 Sv at 26.5 N). The AMOC bias has been one of the hardest challenges for E3SM. The current hypothesis is that AMOC is very sensitive in the coupled system and small perturbations can easily cause it to collapse and that no single change can explain the degradation in AMOC.
Marine Biogeochemistry
The global ocean plays a critical role in the global carbon cycle, having stored an estimated total of 165 ± 20 PgC of excess carbon—or about a quarter of anthropogenic CO2 emissions—since 1750 (Le Quéré et al., 2018). An accurate representation of ocean CO2 uptake, and its response to changes in the dynamics of the coupled climate system, is essential for Earth System Models to predict the response of the climate system to potential future changes in CO2 emissions. Despite decades of research, interactions between Earth’s carbon cycle and climate remain among the largest uncertainties in predicting future climate response to a given magnitude of CO2 emissions (Gregory et al., 2009). In E3SMv1.1-BGC, globally-integrated ocean CO2 uptake begins to diverge from other CMIP models and observational benchmarks (Ciais et al., 2013) in the mid-twentieth century, and is biased low by about 50% by the end of the 20th century (Burrows et al., 2020). The E3SMv1.1-BGC model also substantially underpredicts biomass production, and overpredicts anoxia. The E3SM team is currently investigating several potential causes of these biases, which are currently thought to primarily result from interactions between the ocean physics and ocean biology. The current hypothesis is that these biases are primarily the result of interactions of the biogeochemical processes with parameterized ocean physics processes, including mesoscale eddies, vertical mixing, and freshwater inputs.
Cryosphere
Antarctic and Southern Ocean Climate
Flow of the Antarctic Ice Sheet as simulated by MALI. See Scientific Visualization of E3SM Cryosphere Campaign Simulations
The biggest uncertainty in near future sea level rise (SLR) comes from the Antarctic Ice Sheet (AIS). Work towards coupling of a dynamic AIS model to E3SM is ongoing with the primary piece of uncompleted development being the implementation of wetting-and-drying in the ocean model, which is needed for modeling the advance and retreat of marine ice sheet grounding lines as the ice sheet evolves in response to climate forcing. Other relevant advancements during E3SM phase 2 include: 1) the addition of a new snowpack model, which is appropriate for modeling the evolution of deep snowpacks on ice sheets; and 2) the allowance for ocean circulation within Antarctic ice shelf cavities, including heat and mass exchange between the ocean and overlying ice shelves. Both are in the process of being validated, including characterization of relevant model biases. E3SM simulated sub-ice shelf melt rates are generally realistic for the large, “cold” ice shelves (Ross and Filchner-Ronne) even at low (~30 km resolution). However, even at higher resolutions (~10 km), melt rates under smaller “warm” shelves — those where melting is controlled largely by access of CDW — are generally biased too low. This is because of the importance of small-scale topographic and ocean dynamical features in controlling the flow of CDW into ice shelf cavities. With respect to a realistic representation of Southern Ocean climate in E3SM v2, ocean-and-ice biases of concern include: too-fresh ocean surface; overly active and extensive open-ocean convection at eddying resolutions; too little production of High-Salinity Shelf Water and Antarctic Bottom Water; too strong of an Antarctic Slope Front at eddying resolutions; warm ocean bias at depth in low-resolution simulations and cold ocean bias at depth in eddying simulations; too strong amplitude of the seasonal sea ice ice cycle. It is important to note that many of these same biases are common among “CMIP class” coupled climate models. Additional relevant Southern Ocean climate biases on the atmosphere side include: overly strong zonal winds, both offshore (westerlies) and nearshore (easterlies); overly deep sub-polar lows; excessive surface precipitation along the East Antarctic coast and over the Amundsen and Bellingshausen Seas; near surface warm biases, particularly in the warm season. Other ongoing work under the Cryosphere science campaign includes 1) adding support for using these new science capabilities with historical and future scenarios, 2) support for a dynamic Greenland ice sheet component in E3SM, 3) support for evolving iceberg fluxes coupled to dynamic ice sheet evolution, and 4) support for the coupling of dynamic subglacial discharge into the ocean.
Physics and Numerics for coupling of the various components with high and variable resolutions
Earth System Models like the E3SM are typically constructed by coupling together component models of the atmosphere, ocean, land and ice. Within these large components, there are sub-components, parameterizations and even complex subgrid models that must also be coupled together. This modularized approach in model construction allows developers to focus on areas of expertise and leverage community developments but also presents a number of challenges, particularly as high- or variable-resolution meshes are introduced and a larger range of space and time scales must be represented. With meshes that can focus resolution in different places for each component, there can be large disparities between meshes that vary with location on the globe. Coupling then requires the spatial interpolation of fields in a way that can adapt to local mesh conditions while maintaining high accuracy and preservation of desired properties like energy and water conservation, upper and lower bounds, or vector invariants. Similarly, faster physical timescales and finer spatial scales place limits on the time integration of the coupled solution and can prevent century-scale simulations. Often attempts to mitigate time-step bottlenecks through sub-cycling or concurrent time-lagged integration result in numerical inconsistencies, instability and loss of conservation. Earth System Models require algorithms that can work across component boundaries to advance the coupled solution in time in a consistent, stable and accurate way while still allowing components and subcomponents flexibility in their internal algorithms. As new computing architectures adopt a hybrid approach, the ability to map individual components and subcomponents to appropriate hardware elements and load-balance the resulting system will require new techniques to distribute work across a hybrid system in an efficient manner. Finally, a focus on component development often results in unexpected feedbacks or instabilities that are not detected until the final integration stages. Improved methods for formal numerical analysis of the coupled system or methods for faster early-to-intermediate coupled testing and tuning of new components would detect problems earlier in their development and speed the final integration and evaluation process.
Diagnostics and Analysis Infrastructure
An example of the polar contour plots for precipitation rate created using e3sm-diags
The diagnostics and analysis infrastructure is built to support model analysis cross all of the E3SM computation platforms. Nearly all of the necessary software for post-processing and analyzing E3SM simulations is installed on these supported platforms through a Conda environment: E3SM-Unified. This analysis environment covers diagnostics packages including the E3SM Diagnostics package (e3sm_diags), MPAS-Analysis and ILAMB for components model evaluation. More tools including PMP and other CMEC packages are being evaluated to complement current analysis capabilities. It remains challenging to support multiple fast-paced tool packages and to utilize them due to their usage idiosyncrasies. A recently developed post-processing tool, zppy, supports automating configurable post-processing and analysis tasks that are commonly performed for different science campaigns. Zppy helps standardize documenting simulation results, can facilitate the code review process, and further reduces the learning curve for new developers to run E3SM post-processing. It is critical to use diagnostics tools to monitor the change of model performance in the development cycle. The Infrastructure group works closely with the science groups and collaborators to implement new components – especially those that tackle the aforementioned model biases – to standardize analysis workflow in a timely fashion to influence model development.
