Performance Portability in Flash-X

Research topic and goals

Flash-X is a multiphysics simulation instrument designed for heterogeneous hardware platforms. The code is highly composable and can simulate physical phenomena in several scientific domains. It is derived from FLASH, which has been a community code for several communities over the last 20 years. A language agnostic performance portability solution for the code was designed in this collaboration. The solution includes several tools that are individually relatively simple and easy to maintain by non-experts, but together deliver a comprehensive performance portability solution.

Results for 2022/2023

Flash-X was released in February 2022, with several features of the performance portability layer integrated. The release won an R&D 100 award (Dubey et al. 2022). A paper was presented at Heteropar 2021 on one of the tools, Milhoja, which manages the runtime. (O’Neal et al. 2022) using a stand-alone simplified application.

Results for 2023/2024

This year we worked on various code generation tools that each focus on one aspect of code generation, but are needed to integrate the performance portability layer with various physics. We used the tool CGKit for generated different control flow variants of the primary hydrodynamics solver in Flash-X. The results were put in a paper that was submitted to the FGCS special issue on jlesc. The paper is also on arxiv (Rudi et al. 2024). We were also able to integrate a full-featured hydrodynamics solver with CGKit (for control flow) and Milhoja (for data movement). We are currently exploring performance of the code with these tools.

Visits and meetings

Anshu Dubey (ANL) visited Riken in Feburary 2023. We continued to meet online on a regular basis and when personnel are at the same conferences.

Impact and publications

  1. Rudi, Johann, Youngjun Lee, Aidan H. Chadha, Mohamed Wahib, Klaus Weide, Jared P. O’Neal, and Anshu Dubey. 2024. “CG-Kit: Code Generation Toolkit for Performant and Maintainable Variants of Source Code Applied to Flash-X Hydrodynamics Simulations.” 
    @misc{rudi2024cgkit,
  title = {CG-Kit: Code Generation Toolkit for Performant and Maintainable Variants of Source Code Applied to Flash-X Hydrodynamics Simulations},
  author = {Rudi, Johann and Lee, Youngjun and Chadha, Aidan H. and Wahib, Mohamed and Weide, Klaus and O'Neal, Jared P. and Dubey, Anshu},
  year = {2024},
  eprint = {2401.03378},
  archiveprefix = {arXiv},
  note = {submitted to FGCS},
  primaryclass = {cs.DC}
}
  2. O’Neal, Jared, Mohamed Wahib, Anshu Dubey, Klaus Weide, Tom Klosterman, and Johann Rudi. 2022. “Domain-Specific Runtime to Orchestrate Computation on Heterogeneous Platforms.” In European Conference on Parallel Processing, 154–65. Springer, Cham. 
    @inproceedings{onealEtAl2022,
  title = {Domain-specific runtime to orchestrate computation on heterogeneous platforms},
  author = {O’Neal, Jared and Wahib, Mohamed and Dubey, Anshu and Weide, Klaus and Klosterman, Tom and Rudi, Johann},
  booktitle = {European Conference on Parallel Processing},
  pages = {154--165},
  year = {2022},
  organization = {Springer, Cham}
}
  3. Dubey, Anshu, Klaus Weide, Jared O’Neal, Akash Dhruv, Sean Couch, J Austin Harris, Tom Klosterman, et al. 2022. “Flash-X: A Multiphysics Simulation Software Instrument.” SoftwareX 19. Elsevier: 101168. 
    @article{dubeyEtAl2022,
  title = {Flash-X: A multiphysics simulation software instrument},
  author = {Dubey, Anshu and Weide, Klaus and O’Neal, Jared and Dhruv, Akash and Couch, Sean and Harris, J Austin and Klosterman, Tom and Jain, Rajeev and Rudi, Johann and Messer, Bronson and others},
  journal = {SoftwareX},
  volume = {19},
  pages = {101168},
  year = {2022},
  publisher = {Elsevier}
}

Future plans

Continue to integrate Milhoja and CGKit with more complex physics in Flash-X and continue to explore performance.

References

  1. Dubey, Anshu, and Tom Klosterman. 2022. “Language Agnostic Approach for Unification of Implementation Variants for Different Computing Devices.” In Parallel Processing and Applied Mathematics Conference. 
    @inproceedings{dubey2022macro,
  title = {Language Agnostic Approach for Unification of Implementation Variants for Different Computing Devices},
  author = {Dubey, Anshu and Klosterman, Tom},
  booktitle = {Parallel Processing and Applied Mathematics Conference},
  year = {2022}
}
  2. O’Neal, Jared, Mohamed Wahib, Anshu Dubey, Klaus Weide, Tom Klosterman, and Johann Rudi. 2022. “Domain-Specific Runtime to Orchestrate Computation on Heterogeneous Platforms.” In European Conference on Parallel Processing, 154–65. Springer, Cham. 
    @inproceedings{o2022domain,
  title = {Domain-specific runtime to orchestrate computation on heterogeneous platforms},
  author = {O’Neal, Jared and Wahib, Mohamed and Dubey, Anshu and Weide, Klaus and Klosterman, Tom and Rudi, Johann},
  booktitle = {European Conference on Parallel Processing},
  pages = {154--165},
  year = {2022},
  organization = {Springer, Cham}
}
  3. Dubey, Anshu, Klaus Weide, Jared O’Neal, Akash Dhruv, Sean Couch, J Austin Harris, Tom Klosterman, et al. 2022. “Flash-X: A Multiphysics Simulation Software Instrument.” SoftwareX 19. Elsevier: 101168. 
    @article{dubey2022flash,
  title = {Flash-X: A multiphysics simulation software instrument},
  author = {Dubey, Anshu and Weide, Klaus and O’Neal, Jared and Dhruv, Akash and Couch, Sean and Harris, J Austin and Klosterman, Tom and Jain, Rajeev and Rudi, Johann and Messer, Bronson and others},
  journal = {SoftwareX},
  volume = {19},
  pages = {101168},
  year = {2022},
  publisher = {Elsevier}
}
  4. Harris, J Austin, Ran Chu, Sean M Couch, Anshu Dubey, Eirik Endeve, Antigoni Georgiadou, Rajeev Jain, et al. 2022. “Exascale Models of Stellar Explosions: Quintessential Multi-Physics Simulation.” The International Journal of High Performance Computing Applications 36 (1). SAGE Publications Sage UK: London, England: 59–77. 
    @article{harris2022exascale,
  title = {Exascale models of stellar explosions: Quintessential multi-physics simulation},
  author = {Harris, J Austin and Chu, Ran and Couch, Sean M and Dubey, Anshu and Endeve, Eirik and Georgiadou, Antigoni and Jain, Rajeev and Kasen, Daniel and Laiu, M Paul and Messer, OE B and others},
  journal = {The International Journal of High Performance Computing Applications},
  volume = {36},
  number = {1},
  pages = {59--77},
  year = {2022},
  publisher = {SAGE Publications Sage UK: London, England}
}
  5. Couch, Sean M, Jared Carlson, Michael Pajkos, Brian W O’Shea, Anshu Dubey, and Tom Klosterman. 2021. “Towards Performance Portability in the Spark Astrophysical Magnetohydrodynamics Solver in the Flash-X Simulation Framework.” Parallel Computing 108. North-Holland: 102830. 
    @article{couch2021towards,
  title = {Towards performance portability in the Spark astrophysical magnetohydrodynamics solver in the Flash-X simulation framework},
  author = {Couch, Sean M and Carlson, Jared and Pajkos, Michael and O’Shea, Brian W and Dubey, Anshu and Klosterman, Tom},
  journal = {Parallel Computing},
  volume = {108},
  pages = {102830},
  year = {2021},
  publisher = {North-Holland}
}
  6. Rudi, Johann, Jared O’Neal, Mohamed Wahib, and Anshu Dubey. 2021. “CodeFlow: A Code Generation System for Flash-X Orchestration Runtime.” Argonne National Lab.(ANL), Argonne, IL (United States). 
    @techreport{rudi2021codeflow,
  title = {CodeFlow: A Code Generation System for Flash-X Orchestration Runtime},
  author = {Rudi, Johann and O'Neal, Jared and Wahib, Mohamed and Dubey, Anshu},
  year = {2021},
  institution = {Argonne National Lab.(ANL), Argonne, IL (United States)}
}