v1 Workflow

E3SM Workbench

E3SM Workbench encapsulates underlying infrastructure, such as process flow, analysis and visualization, diagnostics, provenance, and data management

The E3SM Workflow Group has developed and deployed E3SM infrastructure capabilities that facilitate rapid model development and testing. We have leveraged DOE investments in the ESGF (ESGF 2017), the UV-­CDAT (Doutriaux 2017), NCO (NCO 2018), Globus Online (Globus 2018), and ProvEn (Elsethagen 2016) to integrate a flexible framework that provides workflow orchestration, provenance capture and management, simulation analysis and visualization, and automated testing and evaluation capabilities.

The Workflow Group focused on six main developments.

E3SM Workbench and Process Flow

Process Flow is a smart, distributed, and asynchronous job scheduling tool that automatically initiates and executes a wide range of backend services, including the generation of regridded monthly climatologies and time series of model output, multiple diagnostics, managed Globus data transfers, and automated diagnostic sharing through a web viewer. The E3SM Workbench is a user-friendly UI environment that encapsulates and integrates all underlying functionality, the development of it had already started but the full capability will be achieved with v2 model release (see figure).

Data Management

E3SM will release model output from all major simulations and all data that were used for any E3SM publication and will serve the data through ESGF. The Workflow Group has established two ESGF nodes in LLNL and ANL, that support data ingests via Globus transfers. The E3SM SVN repository hosted at ORNL/CADES is used for input data, remapping files, and observational data. Integration with the E3SM Workbench will include automatic publication services and CMOR through ESGF in support of E3SM contribution to CMIP6 with DECK experiments (Eyring et al. 2016a).

Analysis and Visualization

The E3SM post-processing software stack utilizes the UV-CDAT, a Python-based tool designed to run analysis and visualization tools and techniques while capturing independent workflows for enhancing reproducibility. The strengths of this framework include parallel streaming statistics, optimized parallel I/O, remote interactive execution, workflow capabilities, and automatic data provenance capture. In addition to the ability to intuitively add custom functionality, it supports server-side aggregation, subsetting, and regridding. E3SM contributions to UV-CDAT include: multiple easy-to-install official versions via Anaconda open-source distribution; support of the latest regridding by interfacing to the ESMF library; new 1- and 2-dimensional visualizations required by E3SM diagnostics; and documentation and an online tutorial via Jupyter notebooks.


The focus of this work is to generate diagnostics and plots substantially equivalent to the climate communities AMWG and LMWG diagnostics while adding new capabilities of comparison to many different and newer observational data sets, as well as new model diagnostics using the extended UV-CDAT framework called the Community Diagnostics Package (CDP). The E3SM Workflow Diagnostics Team also added significant new and performance-enhancing functionalities to the NCO package, which is critical for E3SM diagnostics efforts and heavily used for generating climatologies and regridding tasks. Other developments include testing and deployment procedures on all E3SM-supported platforms and beginning the integration of external diagnostics packages and libraries such as ILAMB via CIME, ARM diagnostics, and the PCMDI Metrics Package (PMP) (Gleckler 2015, 2016).

Provenance Capture

ProvEn is a comprehensive provenance infrastructure that will not only provide insights explaining which factors contributed to simulation results, but also combine workflow provenance information with performance metrics for anomaly detection. The work during Phase I of the project includes development of the Harvest Provenance Interface (HAPI) that allows CIME workflows and other grid-based provenance to be integrated and searchable via the ProvEn dashboard. The ProvEn prototype for reproducing the E3SM run is nearing completion.

Infrastructure: hardware, software and user support

The Workflow Group supports software installed in each E3SM-supported DOE computational facilities. The group also maintains versioning information, points of contact, and links to simple “how to” pages for each software package. Supported centers and machines include: OLCF – Rhea, LLNL – ACME1/AIMS4, NERSC – Edison/Cori, ANL – Anvil, and ALCF – Cooley.

Send this to a friend