zstash v1.2.0 supports Globus

Zstash, an E3SM long-term archiving tool, now supports the use of Globus.

Zstash, an E3SM long-term archiving tool, now supports the use of Globus to automatically transfer the data from a machine without a High Performance Storage System (HPSS) to the one attached to HPSS storage. Another notable improvement is the ability to extract or check a subset of tar files.

zstash is written entirely in Python using standard libraries. It generates local, standard tar files and then transfers them to HPSS. Checksums (md5) are computed on-the-fly during archiving and verified on extraction. Metadata is stored in a SQLite3 index database, enabling faster retrieval for target files.zstash has parallel extraction and verification for increased performance. All E3SMv1 production simulations are required to be archived using zstash.

The original zstash releases focused on supporting machines with HPSS installed. During Phase 2 of E3SM, the introduction of new computation platforms, including Compy and Chrysalis, required zstash to be more flexible to support local disk archiving and to transfer local archives to platforms with HPSS.

The latest version of stash (v1.2.0) builds on this further by allowing users to automatically transfer data from a machine without HPSS to HPSS on a different machine, using Globus. Users simply specify

--hpss=globus://<Globus endpoint UUID>/<path to archive>

Names alcf and nersc are recognized as referring to the ALCF HPSS and NERSC HPSS endpoints, e.g.globus://nersc/~/my_archive.

Another feature included in this release is the new –tars option, which can be used to specify which tars to check or extract:

# Starting at 00005a until the end
zstash check --tars=00005a-
# Starting from the beginning to 00005a (included)
zstash check --tars=-00005a
# Specific range
zstash check --tars=00005a-00005c
# Selected tar files
zstash check --tars=00003e,00004e,000059
# Mix and match
zstash check --tars=000030-00003e,00004e,00005a-

More information on using these new features can be found at Usage – zstash documentation

Documentation

• Zstash Documentation
  • Zstash documentation – zstash documentation
• Github (source code)
  • GitHub – E3SM-Project/zstash: Long term HPSS archiving tool for E3SM
• Getting  Started
  • Getting started – zstash documentation
• Tutorial
  • Tutorial – zstash documentation
• Other Posts
  • New Zstash Capabilities
  • Zstash HPSS Archive