E3SM Processflow

E3SM Workbench encapsulates underlying infrastructure, such as process flow, analysis and visualization, diagnostics, provenance, and data management

E3SM developed infrastructure capabilities that facilitate rapid model development and testing. We have leveraged DOE investments in the ESGF, the CDAT, NCO, Globus Online, and ProvEn to integrate a flexible framework that provides workflow orchestration, provenance capture and management, simulation analysis and visualization, and automated testing and evaluation capabilities.

Process Flow is a smart, distributed, and asynchronous job scheduling tool that automatically initiates and executes a wide range of backend services, including the generation of regridded monthly climatologies and time series of model output, multiple diagnostics, managed Globus data transfers, and automated diagnostic sharing through a web viewer. The E3SM Workbench is a user-friendly UI environment that encapsulates and integrates all underlying functionality, the development of it had already started but the full capability will be achieved with v2 model release.

What is the processflow?

The acme_processflow is a command line tool to automatically run post processing jobs for ACME model output. The tool takes a single configuration files and runs a series of long running transfer and processing jobs on any amount of model output, running the set of jobs on any number of set lengths. The output doesn’t have to exist before the tool is run, meaning it can be started at the same time as the model and as data is generated will transfer it, and start the processing and diagnostics as soon as the first complete set is available.

Processflow is a python package and it is an open source development project on github that can be easily installed with conda. 

The processflow use Globus to transfer the model output data, once it has the required data it runs the requested jobs at the selected intervals. Current supported job types include E3SM diagnostics, AMWG diagnostics, regridded climatologies, time series variable extraction and A-Prime coupled diagnostics. The plot output is automatically made available via web hosting for easy viewing.