Aiming to be among the First Applications to Run at Exascale

By Scott Gibson

The ExaSky project conducts extreme-scale simulations to further our understanding of the makeup and evolution of the universe. The effort’s Argonne National Laboratory principal investigator, Katrin Heitmann, is a guest on the Let’s Talk Exascale podcast.

“We carry out simulations to understand how structures form in the universe and to create synthetic sky catalogs that will aid the analysis of large cosmological survey data,” Heitmann said. “A fundamental question we are aiming to answer is, what is the origin of the accelerated expansion of the universe?”

ExaSky is an Application Development effort within the Earth and Space Science research agenda of the US Department of Energy’s Exascale Computing Project (ECP). This area addresses fundamental scientific questions, from the origin of the universe and chemical elements to planetary processes and interactions affecting life and longevity.

“We are hoping to be one of the first applications that will successfully carry out science runs on the DOE exascale machines when they arrive [in 2021],” Heitmann said. A key task in achieving that aspiration is preparing the code that will instruct ExaSky, the Hardware/Hybrid Accelerated Cosmology Code (HACC), to be ready to go on the Aurora and Frontier machines at Argonne and Oak Ridge labs, respectively.

Katrin Heitmann of Argonne National Laboratory

Katrin Heitmann of Argonne National Laboratory

The ExaSky team currently is running HACC at scale on available pre-exascale hardware—like Oak Ridge’s Summit supercomputer, for example—to prepare HACC to address ExaSky’s challenge problem. The code has to be ready for relevant science implementations, such as in situ analysis and overall analysis capabilities, and for continued scaling runs to meet performance standards called figures of merit.

Using its code and the larger-scale machines, ExaSky is conducting simulations that tackle different physics challenges: one for gravity only, and the other for hydrodynamics at unprecedented scale and resolution.

“Running a hydrodynamics simulation basically means that we add more physics to the simulation that is particularly relevant on small scales,” Heitmann said. “These simulations will be essential to aid the analysis of upcoming cosmological survey results as obtained from the ground and space, including results from the Large Synoptic Survey Telescope, the Dark Energy Spectroscopic Instrument, the Wide Field Infrared Telescope, and SPHEREx. The simulations will be the base for detailed synthetic sky catalogs.”

The main difficulty ExaSky faces is the uncertainty as to the programming models available on the different machines. To deal with that, the ExaSky team is collaborating very closely with the computing system companies and taking advantage of the available testbeds.

“Our current challenge problem is designed to run across the full machine [on both Aurora and Frontier], and doing so on a new machine is always difficult,” Heitmann said. “We know from experience, having been first users in the past on Roadrunner, Mira, Titan, and Summit; and each of them had unique hurdles when the machine hit the floor.”

ExaSky’s major successes suggest a readiness to overcome obstacles, whether identified or not.

“We were able to immediately carry out large science runs on Summit when the machine was available, and we are on a good trajectory in terms of our figure of merit,” Heitmann said. “We have shown excellent scaling results on Summit for HACC up to the full machine and we already have a version of HACC available that will be able to run on both Aurora and Frontier.”

The expected impact of the team’s considerable preparation is the capability to perform cosmological simulations at unprecedented scale, she added. The simulations will allow the team to examine the impact of baryonic physics on a scale relevant to dark energy studies, as well as address multiple cosmic probes.

As the ExaSky effort moves forward, the team will continue to prepare for the new hardware and software that will be available on Aurora and Frontier.

“For the gravity-only simulation, we are mostly refining our in situ analysis tool suite and are developing solutions for reducing the large amount of data that can be generated on the exascale system,” Heitmann said. “We are also implementing solutions that will enable us to deal with possible short mean time between failures in an efficient way. For the hydrodynamics simulations, we are still optimizing the code and are working on some of the physics implementations.”