Recent News
School of Engineering students winners at Lobo Hackathon
April 25, 2024
Two from School of Engineering to receive local 40 Under 40 awards
April 18, 2024
Water Science Communication Fellowship lets undergraduates get feet wet in research
April 17, 2024
Alumnus endows ‘station of innovation’ parking space for the School of Engineering
April 15, 2024
News Archives
Mechanical engineering doctoral candidate develops open-source GPU-capable fluid dynamics code
January 27, 2022 - Melanie Furber Fudge, UNM Center for Advanced Research Computing
FIESTA results are shown from the Shock Tube Lab shock-driven Kelvin-Helmholtz instability experiment: Vertical and horizontal cross section of a dense gas column at different times during interaction with a shock wave.
When Brian Romero began his doctoral research in the Advanced Fluids Lab, led by Professor Svetlana Poroseva at The University of New Mexico’s Department of Mechanical Engineering, he quickly realized that the tool he needed to complete his work had not yet been developed.
He needed a computational fluid dynamics solver capable of simulating supersonic flows with shock waves but “couldn’t find one that I liked, that was fast enough, or that was open source and available to me,” Romero said.
With oversight from his advisor, Poroseva, and with partial support from the Defense Threat Reduction Agency (DTRA) and Los Alamos National Laboratory (LANL) contracts, Romero turned to UNM’s Center for Advanced Research Computing (CARC) and set out to construct the necessary tool.
“CARC was the only cluster I had access to when I first started coding,” Romero said. “I was lucky enough to have the CARC Xena GPU [graphics processing unit] cluster to work with. I couldn’t have developed and tested without that cluster. FIESTA© [Fast Interface Evolution, Shocks and Transition in the Atmosphere] would not have been possible without CARC. CARC is where FIESTA© was born.” When Romero designed FIESTA©, he designed it with the future in mind. Thinking beyond the use for his own work, he wanted to develop code that was user and developer-friendly and could accommodate researchers’ ever-evolving use of GPUs, high-performance computing (HPC) systems and exascale supercomputers.
“FIESTA© is a product of my Ph.D. It is a tool that I designed and built out of necessity to get my work done,” Romero said. “I wanted to make it easy for users and future developers to use. I designed it in a modular way, so other types of physics can be easily implemented in it. It is designed to do more than it can do now.”
Although initially designed for atmospheric simulations, as well as some smaller laboratory-scale experiments, this future-thinking built into the code allows FIESTA© to do a wide range of simulations.
“I designed FIESTA© for exascale systems with neat technology built in to enable that. I used a framework called Kokkos, from Sandia Labs. Kokkos is a performance portable ecosystem, which enables me to run FIESTA© on any GPU machine. Another popular GPU technology, CUDA®, can also get a code running on GPUs, but is limited to NVIDIA® architectures. Kokkos lets me run FIESTA© on GPUs from a variety vendors supported by the Kokkos team, while still being able to run FIESTA© on traditional CPUs [central processing unit]. This makes the code extremely flexible. [In the past] we would need to have two codes: one to run on GPUs and one to run on CPUs. But now with Kokkos, we have one code that can run anywhere. And Kokkos has long-term support from the Department of Energy, so FIESTA© will continue to run on new GPUs that come out in the future. This makes FIESTA© ready for the next generation of exascale hardware.”
FIESTA© also has advanced features for parallel file systems using HDF5® and a unique input file system using an embedded scripting language called Lua.
“I haven’t seen Lua in other CFD [computational fluid dynamics] codes, this is a really useful feature I’m proud of,” Romero said.
Romero is working closely with UNM’s Shock Tube Lab, an experimental group in UNM’s mechanical engineering department led by Professor Peter Vorobieff, to verify data and check the accuracy of FIESTA©. Recently, FIESTA© simulations were compared to shock-tube experiments to verify the mechanism behind the shock-driven Kelvin-Helmholtz instability, a phenomenon useful for mixing fuel and air in certain types of scramjet engines. Romero explains “past experiments were not able to reveal necessary details to confidently explain this mechanism. Simulations with FIESTA© were able to confirm this mechanism.”
This work, done in collaboration with Poroseva, Vorobieff and LANL scientist Jon Reisner, has been published in the Physics of Fluids Journal and has been presented at five conferences in the last two years.
“…FIESTA© was run on GPU clusters at Los Alamos National Laboratory and Lawrence Livermore National Laboratory,” Romero said. “Running simulations on GPUs produced simulation results over ten times faster than if we ran on a CPU cluster … Since we could run so quickly on GPUs, we were able to do multiple simulations to explore the effects of different parameters on the shock-driven Kelvin-Helmholtz instability.”
Romero and Poroseva named the code FIESTA© in recognition of the connection to New Mexico; the chile pepper logo is a nod to Romero’s love of New Mexican cuisine.
FIESTA© is a GPU capable high-order compressible flow solver designed by Romero and Poroseva and is owned and copyrighted by UNM. Although Romero remains the primary user, he notes others have started to pick it up and use it, including others in his department as well as in UNM’s Predictive Science Academic Alliance Program (PSAAP).
“FIESTA© is a great example of modern scientific applications design in both its state-of-the-art numerical algorithms and its use of Kokkos,” said Patrick Bridges, CARC director and principal investigator with PSAAP. “As a result, it’s been a great example for us to use to understand and optimize the communication requirements of next-generation high-performance applications.”
Although Romero plans to graduate this summer, his vision is for FIESTA© to remain a UNM asset for years to come.
“We are looking for students to continue to develop FIESTA©,” he said. “It has the potential to be a problem-solving code for researchers from anywhere, not just UNM.”
FIESTA© will be made available publicly as open-source software in spring 2022. Meanwhile, Romero continues to utilize CARC resources as he finishes his doctoral requirements.