Chrysalis – ANL
E3SM Dedicated Machine - Chrysalis
Chrysalis, a dedicated E3SM machine, was purchased by the Department of Energy (DOE) Office of Biological and Environmental Research (BER) at the end of 2020, and installed at Argonne National Laboratory (ANL).
Chrysalis resides next to Anvil, an older E3SM machine, and they now share a new large (3 PB) disk space and a 4 PB tape backup system. Chrysalis is set up similarly to Anvil, with shared home directories and accounts, which enables users of Anvil to seamlessly transition to work on the new machine. Chrysalis has updated versions of Intel and GNU compilers.
Hardware
- 512 nodes, dual socket, 64 cores per node
- AMD Epyc 7532 processors (32 cores per CPU, 2.4 GHz)
- 256 GB 16 channel DDR4 3200 memory per node
- HDR200 interconnect
- 3 PB of disk space (shared with Anvil)
- 4 PB tape backup
- 2 dedicated login nodes, dual socket, 128 cores per node, AMD Epyc 7702 2.0 GHz
Capabilities
- Chrysalis will provide 4.4 M node-hours every calendar year (2.2 times more than the E3SM Compy allocation)
- Based on early v2 cost estimates (100 node-hours per simulated year), the 4.4 M node-hours/calendar year is about 45,000 years of Water Cycle coupled low-resolution simulations (~100 km horizontal atmosphere res.), or 4,500 years of Continental US (CONUS) Regionally Refined Mesh (RRM) simulations, or 700 years of high-res (25 km atmosphere) simulations.
Allocations
Chrysalis nodes are open to all developers working on tasks funded by the E3SM project. It is not open to work on the E3SM model funded by other projects such as SciDAC and CMDV.
- E3SM – 100 % allocation
Getting an Account
For information on how to get an account and quick start, E3SM members can access the internal Confluence website at:
- Chrysalis – E3SM’s 1PF AMD system (internal Confluence page)
Fun Fact
The name “Chrysalis” was voted on by the leadership team and proposed by the E3SM Program Manager, Dr. Xujing Davis. It reminds her of her experience of raising Monarch Butterflies. “Watching the process of how a caterpillar phased into a beautiful butterfly via the Chrysalis is amazing; it made me think of our E3SM development/transformation from one phase to another and the incredible effort behind the scenes from everyone through the process. After systematically consuming leaves/energy, the chrysalis provides a protective housing for a caterpillar to undergo a remarkable, transformative process over time, culminating in its evolution into a magnificent butterfly. It is our hope and plan that this new machine act in similar fashion for the development of DOE’s E3SM model.”, explained Dr. Davis.
