One Year After Model Release, Developers Envision a Bigger, Better and Higher-Resolution E3SM

It was just a little more than 12 months ago that four years of hard work culminated with the official release of the Energy Exascale Earth System Model (E3SM) to the scientific community.

The U.S. Department of Energy (DOE) introduced E3SM as the “first end-to-end multi-scale model” to build an understanding of complex and changing earth systems—information that can help strengthen the resilience of people, our economy, energy resources, and national security.

The development of E3SM version 1 closed on April 24, 2018, with a code release that offered scientists a new set of tools to simulate the atmosphere, land, ocean, and sea ice. Software packages were painstakingly developed by top geophysical and computational scientists from eight DOE national laboratories, providing the ability to forecast how earth systems interact today and into the future.

The past year has witnessed a dizzying amount of E3SM activity, both from within DOE and from the E3SM research community. Here are just a few major accomplishments:

Successful Project Review: A critical milestone project review took place in May 2018 before a rigorous panel with tough science questions. Ultimately, the panel recommended the E3SM project to move forward to the next phase of development. “The review went very well,” said E3SM Chief Scientist Ruby Leung of Pacific Northwest National Laboratory. “The committee was enthusiastic about our planned capability to go down to much, much higher resolution and encouraged us to develop metrics to track progress towards improved capabilities to model the water cycle, biogeochemistry, and cryosphere systems to support DOE’s mission.”

Simulation Campaign Progress: Simulations for the water cycle, biogeochemistry and the cryosphere made significant steps forward over the past year, including nearly finishing all water cycle version 1 simulations at standard resolution (get a detailed update here).

• The water-cycle version 1 campaign is winding down, with ongoing high-resolution simulations moving forward. A landmark achievement was an overview paper in the Journal of Advances in Modeling Earth Systems that describes and evaluates the performance of the model in the water cycle configuration. About 3,000 years of simulations have been performed to date.
• The biogeochemistry campaign, focused on interactions between biogeochemical cycles and earth system components, is continuing. Work on the simulation campaign had produced 2169 years of control and historical simulations.
• The cryosphere campaign, meanwhile, is ramping up. Simulations from control runs are complete, and investigators are working on upcoming experiment configurations.

CMIP6 DECK Simulations: E3SM scientists published the publish Coupled Model Intercomparison Project Phase 6 (CMIP6), Evaluation, and Characterization of Klima (DECK) water cycle simulations. These are free to download via the Earth System Grid Federation (ESGF) distributed and federated archives.

 Ecosystem Projects Grow in Scope and Funding: In July, eight new projects received awards under DOE’s Scientific Discovery through Advanced Computing (SciDAC) program. Two projects are large, five-year projects, and six focus on improving representations of the atmospheric, land, or sea-ice components in E3SM. Six months later, DOE’s Office of Biological and Environmental Research awarded 13 new research projects a total of $10 million to use or further develop E3SM.

 SCREAM’ing Toward Exascale Modeling: The computer science that will prepare E3SM for an exascale computing future took a leap forward this past year. As E3SM moves toward high-resolution, it must resolve both very large- and very small-scale events with greater detail and at a higher speed. To address this need, E3SM Chief Computational Scientist Mark Taylor of Sandia National Laboratories is working with Peter Caldwell of Lawrence Livermore National Laboratory on a team tasked with next-generation development of E3SM.

“There is a lot of excitement about the launch of a new atmosphere model called SCREAM (Simple Cloud-Resolving E3SM Atmosphere Model) that targets cloud-resolving resolution at 3-kilometer resolution,” said Taylor, noting that the SCREAM project was built on landmark success the computational team had in improving graphics processing performance. “We expect that SCREAM will first be used for 10-year simulations that can look at climate sensitivity with a cloud-resolving model.”

E3SM Support and Project Visibility: As the model was developed, program support activities were also implemented.  A dedicated website (e3sm.org) was launched to provide a ready source of code updates, news, online training, and other important information. The quarterly Floating Points newsletter was created to keep researchers up to date. DOE enhanced computing resources, and brought the E3SM community together with the Integrated Modeling Principal Investigator Meeting and the 2019 E3SM Spring Meeting.

Meanwhile, researchers published dozens of E3SM-related papers, and presented at major meetings, including the 2018 American Geophysical Union (AGU) Fall Meeting and the 2019 American Meteorological Society (AMS) Annual Meeting. Atmospheric and earth system scientists who knew little about E3SM a year earlier are now well aware of this cross-cutting, multi-institutional effort.

Future-Focused Work for E3SM

“Version 3 and 4 (of E3SM) will be quite different from what we have today,” Leung predicted about the model’s future, emphasizing that E3SM is designed for the DOE exascale computing platforms that will soon be available.

“We are working to build a  remarkable capability of global atmospheric modeling at a 3-kilometer resolution,” Leung said.  “Our current high-resolution of 25-kilometers is already resolving important earth system features, but our biggest challenge is that even at this resolution, cumulus clouds—which constitute major uncertainties in earth system models—are still parameterized.”

Reflection of sunlight by clouds is critical to climate, but isn’t captured by conventional General Circulation Models (GCMs). SCREAM aims to explicitly resolve deep convection (the smooth-textured anvil-shaped cloud in the picture) while parameterizing the smaller cumuli.

With the experience gained from the first phase of development focusing on component models, major activities in E3SM phase 2 are re-organized by core groups based on ‘science drivers’ with a more holistic, cross-disciplinary approach to the development of the coupled earth system. These drivers include the water cycle, biogeochemistry, and cryosphere systems. Two additional core groups are organized by computational performance and infrastructure.

Meanwhile, in parallel, development teams are focusing on five next-generation development projects for version 3 and version 4. They are targeting new capabilities to enhance E3SM. These include the effort to achieve 3-kilometer resolution, improving the representation of atmosphere and land models, software engineering, modeling coastal processes, and modeling dynamic ice sheets.

“These are all really exciting developments for E3SM that implement our strategy for high-resolution modeling and representing interactions between human and earth systems,” said Leung. “Versions 3 and 4 will probably not be ready for six years, but starting now will position us well for achieving our vision.”

Project Origins and Energy Focus

The project to develop a new earth system model began in 2014, starting from the high bar set by the Community Earth System Model (CESM). The focus was high-resolution, high-performance computing and science needed for energy planning.

With the ability to model at 25-kilometer-wide grid cells globally or within regionally refined areas, the model’s representations of earth systems would help scientists to better understand the Earth today and in the future.

Researchers designed E3SM not only to improve the predictability of changing earth systems but to evaluate how those changes might stress energy infrastructure. The goal is to support DOE’s mission to plan for robust, efficient, and cost-effective energy infrastructures now and into the distant future.

In an article published by DOE just months after the E3SM launch, E3SM Principal Investigator and Council Chair David Bader of Lawrence Livermore National Laboratory explained the important capability that E3SM could provide to understand earth system interaction with energy infrastructure, production, and use. “DOE and its predecessors have always had a mission to understand the environmental consequences of energy production and use,” he said. “We know that the earth system is not stationary. It changes. We need a way to understand its interactions with human systems that influence its future paths.”

Now under a state of consistent refinement and growing capabilities, E3SM is helping researchers demonstrate the interaction of individual systems—atmosphere, land, ocean, ice, and human activities.