E3SM Quick Start

Get the Code

The E3SM code is available from Github. We only support obtaining the code with “git clone“.  You need at least version 2.0 or later of git. For access to all code versions, add an ssh key to github.  We will not be making .tar.gz or .zip files available at this time.

Open a terminal and issue the following commands:

>git clone -b <ref> --recursive https://github.com/E3SM-Project/E3SM.git
where <ref> is one of the following:
  • maint-1.0
  • maint-1.1
  • maint-1.2
  • maint-2.0
  • maint-2.1

>cd E3SM

NOTE: Do not check out master unless you want to be on your own. We do not support general users on the master branch.

NOTE:  do not use .tar.gz or .zip files Github makes available because they do not include the code from git submodules.

 

Input Data

E3SM model requires several input data files for initial and boundary conditions.

On supported platforms, all the input data for supported cases is already available in world-readable local directories.

For un-supported machines, you should NOT try to download all the inputdata at once.  The CIME case control system will download each missing file for each case you try to run.

Input data is kept on the following servers

E3SM-authored input data lives on the a world-readable http server at

Some cases may require CESM data, which can be obtained from the CESM input data server, which is now world-readable without any authentication required

The CIME case control system will automatically check each server for any missing input data files and download from the first one it finds.

Running E3SM

Running E3SM on the supported NERSC Cori-KNL machine is straightforward. For instance, in order to run CMIP6 pre-industrial control simulation (piControl) (v1 1deg CMIP6) , a scientifically validated configuration, you should follow the steps below, either using the one-step run_e3sm script or using the CIME Case Control System directly.

Running with run_e3sm script

Instructions for E3SM v1.0

Download the script for piControl (run_e3sm.DECKv1b_piControl.ne30_oEC.cori-knl.csh) to NERSC Cori and make it executable:

ssh cori.nersc.gov
wget https://web.lcrc.anl.gov/public/e3sm/run_e3sm.DECKv1b_piControl.ne30_oEC.cori-knl.csh
chmod +x run_e3sm.DECKv1b_piControl.ne30_oEC.cori-knl.csh

If you want to keep all the default settings (a reasonable first choice), you can then simply execute it:

./run_e3sm.DECKv1b_piControl.ne30_oEC.cori-knl.csh

This script will automatically perform the following steps:

  • Clone the source code from the E3SM github repository.
  • Build the executable.
  • Prepare all input files.
  • Submit a 5-day test simulation using 31 nodes to the debug queue.

By default, the source code will be downloaded in the home file system:

~/E3SM_code/20180420

and the executable and output files will be on CSCRATCH:

/global/cscratch1/sd/$USER

You can easily customize the locations by changing the following lines in the script:

set code_root_dir = ~/E3SM_code
set e3sm_simulations_dir = /global/cscratch1/sd/$USER/E3SM_simulations
set case_build_dir = ${e3sm_simulations_dir}/${case_name}/build
set case_run_dir = ${e3sm_simulations_dir}/${case_name}/run
set short_term_archive_root_dir = ${e3sm_simulations_dir}/${case_name}/archive

The script is configured to run a short 5-day test with the default PE layout (31 nodes). This is also easily customizable within the script. To change simulation length and restart file frequency:

set stop_units = ndays
set stop_num = 5
set restart_units = $stop_units
set restart_num = $stop_num

To change PE layout (number of processors)

set processor_config = S

S corresponds to small (31 nodes) and L large (125 nodes).

Running manually

If you prefer, you can also configure, build and run the model using basic CIME commands.

First, obtain the code from github as above. Then execute the following:

cd E3SM/cime/scripts
> ./create_newcase --case cmip6-picontrol --compset A_WCYCL1850S_CMIP6 --res ne30_oECv3_ICG --mach cori-knl --user-mods-dir ../config/e3sm/testmods_dirs/allactive/v1cmip6/
> cd cmip6-picontrol
> ./case.setup
> ./case.build
> ./case.submit

Verifying your results

Once you have successfully completed a test simulation and before you go any further, it is very important that you verify that you have exactly reproduced the original piControl simulation (bit-for-bit or BFB). The easiest way to do this is by looking at the atmosphere global integrals that are printed in the atm log file every single time step.

–NOTE:  VERIFICATION CURRENTLY NOT WORKING —
cd /global/cscratch1/sd/$USER/E3SM_simulations/20180420.DECKv1b_piControl.ne30_oEC.cori-knl
gunzip -c run/atm.log.*.gz | grep '^ nstep, te ' > atm_global.txt

Open atm_global.txt with your favorite editor. There is one line for each time step with the time step number followed by four floating point numbers. These numbers correspond to the global means of input and output total energies for that timestep, the heating rate that is required to explain that energy change, given the atmospheric mass present at that timestep (which changes depending on the amount of water in the atmosphere).

Below is a sample of global mean values that your simulation should exactly match:

nstep, te 0 0.33316446678695908E+10 0.33316446678695908E+10 0.00000000000000000E+00 0.98499926375770388E+05
nstep, te 1 0.33313929836976542E+10 0.33314404836213779E+10 0.26274155501810853E-02 0.98499655017283745E+05
nstep, te 2 0.33314550910159287E+10 0.33314497295562820E+10 -0.29656356589506991E-03 0.98499904536603222E+05
nstep, te 3 0.33314503705129209E+10 0.33314491073043914E+10 -0.69873120085907472E-04 0.98499828883020658E+05
nstep, te 4 0.33314170617464027E+10 0.33314220453280387E+10 0.27566205679904304E-03 0.98499754413300907E+05
nstep, te 5 0.33314045323980412E+10 0.33314079848029923E+10 0.19096642576315332E-03 0.98499783100913352E+05
nstep, te 6 0.33313962051137180E+10 0.33313985181402597E+10 0.12794279309404400E-03 0.98499800722696236E+05
nstep, te 7 0.33313984937634706E+10 0.33313989170515375E+10 0.23413756274133810E-04 0.98499833193543353E+05


[...]
nstep, te 237 0.33326425274001422E+10 0.33326497811551204E+10 0.40122716991089860E-03 0.98501560186958115E+05
nstep, te 238 0.33326085874152384E+10 0.33326158935083303E+10 0.40412220047429643E-03 0.98501547786024210E+05
nstep, te 239 0.33325751781990986E+10 0.33325823864907975E+10 0.39871256286111380E-03 0.98501534323767497E+05
nstep, te 240 0.33325416284682307E+10 0.33325490727624450E+10 0.41176662213212261E-03 0.98501524390200546E+05
nstep, te 241 0.33325140848289461E+10 0.33325204298006172E+10 0.35095975397538318E-03 0.98501520114556770E+05

More simply, you can also just check the md5 hash of the first 240 lines in the file

cat atm_global.txt | head -n 240 | md5sum

And you should obtain the following value for the hash:

44f10337667ee9bb4e38179a24138aae

Troubleshooting

Use github to post a question or report a bug (it requires a github account):

 

 

 

Send this to a friend