E3SM Quick Start

Get the Code

The E3SM code is available from Github.  You do not need a github account to download the 1.0 source code. We only support obtaining the code with “git clone“.  You need at least version 2.0 or later of git. We will not be making .tar.gz or .zip files available at this time.

Open a terminal and issue the following commands:

>git clone -b maint-1.0 https://github.com/E3SM-Project/E3SM.git
>cd E3SM
>git submodule update –init

NOTE:  do not use .tar.gz or .zip files Github makes available because they do not include the code from git submodules.

 

Input Data

E3SM model requires several input data files for initial and boundary conditions.

On supported platforms, all the input data for supported cases is already available in world-readable local directories.

For un-supported machines, you should NOT try to download all the inputdata at once.  The CIME case control system will download each missing file for each case you try to run.

Input data is kept on the following servers

E3SM-authored input data lives on the a world-readable http server at

Some cases may require CESM data, which can be obtained from the CESM input data server, which is now world-readable without any authentication required

The CIME case control system will automatically check each server for any missing input data files and download from the first one it finds.

Running E3SM

Running E3SM on the supported NERSC Edison machine is straightforward. For instance, in order to run CMIP6 pre-industrial control simulation (piControl), a scientifically validated configuration, you should follow the steps below, either using the one-step run_e3sm script or using the CIME commands directly.

Running with run_e3sm script

Download the script for piControl (run_e3sm.DECKv1b_piControl.ne30_oEC.edison.csh) to NERSC Edison and make it executable:

ssh edison.nersc.gov
wget https://e3sm.org/wp-content/uploads/2018/04/run_e3sm.DECKv1b_piControl.ne30_oEC.edison.csh_.txt -O run_e3sm.DECKv1b_piControl.ne30_oEC.edison.csh
chmod +x run_e3sm.DECKv1b_piControl.ne30_oEC.edison.csh

If you want to keep all the default settings (a reasonable first choice), you can then simply execute it:

./run_e3sm.DECKv1b_piControl.ne30_oEC.edison.csh

This script will automatically perform the following steps:

  • Clone the source code from the E3SM github repository.
  • Build the executable.
  • Prepare all input files.
  • Submit a 10-day test simulation using 39 nodes to the debug queue.

By default, the source code will be downloaded in the home file system:

~/E3SM_code/20180420

and the executable and output files will be on CSCRATCH:

/global/cscratch1/sd/$USER

You can easily customize the locations by changing the following lines in the script:

set code_root_dir = ~/E3SM_code
set e3sm_simulations_dir = /global/cscratch1/sd/$USER/E3SM_simulations
set case_build_dir = ${e3sm_simulations_dir}/${case_name}/build
set case_run_dir = ${e3sm_simulations_dir}/${case_name}/run
set short_term_archive_root_dir = ${e3sm_simulations_dir}/${case_name}/archive

The script is configured to run a short 10-day test with the small (S) PE layout (39 nodes). This is also easily customizable within the script. To change simulation length and restart file frequency:

set stop_units = ndays
set stop_num = 10
set restart_units = $stop_units
set restart_num = $stop_num

To change PE layout (number of processors)

set processor_config = S

S corresponds to small (39 nodes), M medium, and L large (285 nodes).

Running manually

If you prefer, you can also configure, build and run the model using basic CIME commands.

First, obtain the code from github as above. Then execute the following:

cd E3SM/cime/scripts
> ./create_newcase --case cmip6-picontrol --compset A_WCYCL1850S_CMIP6 --res ne30_oECv3_ICG --mach edison --user-mods-dir ../config/e3sm/testmods_dirs/allactive/v1cmip6/
> cd cmip6-picontrol
> ./case.setup
> ./case.build
> ./case.submit

Verifying your results

Once you have successfully completed a test simulation and before you go any further, it is very important that you verify that you have exactly reproduced the original piControl simulation (bit-for-bit or BFB). The easiest way to do this is by looking at the atmosphere global integrals that are printed in the atm log file every single time step.


cd /global/cscratch1/sd/$USER/E3SM_simulations/20180419.DECKv1b_piControl.ne30_oEC.edison
gunzip -c run/atm.log.*.gz | grep '^ nstep, te ' > atm_global.txt

Open atm_global.txt with you favorite editor. There is one line for each time step with the time step number followed by four floating point numbers. These numbers correspond to the global means of input and output total energies for that timestep, the heating rate that is required to explain that energy change, given the atmospheric mass present at that timestep (which changes depending on the amount of water in the atmosphere).

Below is a sample of global mean values that your simulation should exactly match:

nstep, te 0 0.33316446678695908E+10 0.33316446678695908E+10 0.00000000000000000E+00 0.98499926375770388E+05
nstep, te 1 0.33313929828608332E+10 0.33314404830534167E+10 0.26274304217767433E-02 0.98499655022749634E+05
nstep, te 2 0.33314550913851910E+10 0.33314497302084837E+10 -0.29654791535514382E-03 0.98499904549659492E+05
nstep, te 3 0.33314503730726762E+10 0.33314491097486286E+10 -0.69879509852037704E-04 0.98499828885826631E+05
nstep, te 4 0.33314171046349101E+10 0.33314220802601190E+10 0.27522195437353644E-03 0.98499754507172227E+05
nstep, te 5 0.33314046372553129E+10 0.33314080662649317E+10 0.18967233603790019E-03 0.98499782917237026E+05


[...]
nstep, te 237 0.33326070660964308E+10 0.33326144990378265E+10 0.41113860195330439E-03 0.98501539040588614E+05
nstep, te 238 0.33325719902090282E+10 0.33325794439750524E+10 0.41229054043555910E-03 0.98501523528842226E+05
nstep, te 239 0.33325366000523758E+10 0.33325440904168444E+10 0.41431498601270198E-03 0.98501505949082362E+05
nstep, te 240 0.33325011443193493E+10 0.33325088716025581E+10 0.42741974241294200E-03 0.98501493116001759E+05
nstep, te 241 0.33324711875206785E+10 0.33324779463772259E+10 0.37385311835612937E-03 0.98501486311916859E+05

More simply, you can also just check the md5 hash of the first 240 lines in the file

cat atm_global.txt | head -n 240 | md5sum

And you should obtain the following value for the hash:

6ccfdd276f22ad2f14332434f5533edf

Send this to a friend