Globus
Transferring data (for users)
The Globus Transfer (http://globus.org) is a recommended service for data transfer between:
- E3SM compute resources (ALCF Mira/Cooley, LCRC Anvil/Cooley/Cetus, LLNL Greyworm, NERSC Edison/Cori, OLCF Titan/Rhea)
- ESGF data nodes (aims3.llnl.gov, esgf.anl.gov)
- data transfer nodes (LLNL DTNs)
- HPPS storage systems
Documentation for both users and developers about the Globus service can be found in ‘Resources’ at the Globus website, http://globus.org. A short user guide covering main aspects of the Globus service from the E3SM project’s perspective is at Mass file transfer – user guide.
The complete list of all clusters, ESGF data nodes, HPSS storage systems and DTN nodes with their Globus endpoint names can be found at Globus Endpoints. The same list of all systems used in the E3SM project can be found at Supported Software, Machines and POCs. Accessibility of the systems through the Globus Transfer service is monitored by the E3SM monitoring system and results of daily tests displayed at https://esgf.anl.gov/monitor/e3sm/.
Publishing data (for users)
Data sets can be published using command line tools (esg-publish) executed on an ESGF data node but also using the ACME Web UI (https://acme.globuscs.info). The publication process is described at Publish v0.3 AMIP data to ESGF and at ESGF Publication with Globus Web Services.
Setting up a Globus endpoint (for admins)
The complete documentation describing how to set up a Globus endpoint is at Globus Connect Server Installation Guide. Admins of ESGF data nodes install Globus Connect Server as a part of the ESGF data node installation. However, setting up additional ESGF data transfer nodes (DTN) is a more complicated process that requires manual configuration. The process is documented at Globus Connect Server on ESGF DTN nodes – admin guide.
Quick Links:
- The Globus Transfer service – http://globus.org
- The Globus documentation fo users, developers, providers – https://docs.globus.org
- A list of all E3SM Globus endpoints – Globus Endpoints
- A list of all E3SM machines – Supported Software, Machines and POC
- A short document covering the most important aspects of the Globus service – Mass file transfer – user guide
- An admin guide about setting up an ESGF DTN – Globus Connect Server on ESGF DTN nodes – admin guide
- The Publication Service (Web UI) – http://acme.globuscs.info
- Documentation about the Publication Service Publish v0.3 AMIP data to ESGF
- Screencast of the webinar about the Publication service ESGF Publication with Globus Web Services
