Advanced data and visualization pipelines on the example of nekRS

Research topic and goals

Fifteen years since the publication of “Exascale computing study: Technology challenges in achieving exascale systems” (Bergman et al. 2008) we have finally arrived at the dawn of the Exascale era. In the US, Aurora, Argonne National Laboratory’s (ANL) first exascale computer is being built, while Europe’s Jülich Supercomputing Centre (JSC) is gearing up for a 2024 launch of JUPITER. However, with increased computing power comes an influx of data that requires novel methods for data analysis, storage, processing, and visualization. The exascale era presents one significant challenge in the form of I/O bandwidth limitations, making reading and writing to disks slower than running simulations. Frameworks like SENSEI (Ayachit et al. 2016) address this issue by providing in situ visualization and analysis tools to process data while it is still in memory.

NekRS (Fischer et al. 2022), an open-source Navier Stokes solver based on the spectral element method targeting classical processors and accelerators like GPUs, is an example of a simulation code that scientists will run on the entire machine. ANL and JSC are already collaborating on experimenting with and improving the corresponding data and visualization pipeline. Please note that while NekRS will be our initial focus, the developed methods and pipeline will be general enough for use with other exascale simulation codes and in situ frameworks.

Our team’s current implementation uses a SENSEI-instrumented NekRS code to stream data in transit via an ADIOS n:m bridge to a ParaView server that manages the visualization. The data is then streamed to the cloud via TRAME, thus making it accessible in the browser and through JupyterLab. This preliminary work has already been performed and is very promising.

This project aims to improve the data and visualization pipeline in four steps: First, the workflow, as mentioned above, will be hardened and thoroughly evaluated. Second, we will evaluate our solution in terms of scaling and throughput. Third, we will perform scaling studies on exascale machines. Finally, we will demonstrate the usage of the pipeline with other simulation codes.

Our goal is to deliver a high-performance, flexible, and user-friendly data and visualization pipeline.

Results for 2022/2023

None yet.

Results for 2023/2024

We have examined the developments in relation to different visualization strategies and at scale. We detailed our approach of instrumenting NekRS, a GPU-focused thermal-fluid simulation code employing the spectral element method (SEM), and analyzed varied \textit{in situ} and \text{in transit} strategies for data rendering. Additionally, we provided concrete scientific use-cases and reported on runs performed on Polaris, Argonne Leadership Computing Facility’s (ALCF) 44 Petaflop supercomputer and Jülich Wizard for European Leadership Science (JUWELS) Booster, Jülich Supercomputing Centre’s (JSC) 71 Petaflop High Performance Computing (HPC) system, offering practical insight into the implications of our methodology. Our results were presented at the ISAV23, awarded with the best paper award, and publish in the ISAV23 proceeding (Mateevitsi et al. 2023).

Visits and meetings

  • Mathis Bode visited ANL in April 2023 to coordinate better on the visualization strategies.

Impact and publications

  1. Mateevitsi, Victor A., Mathis Bode, Nicola Ferrier, Paul Fischer, Jens Henrik Göbbert, Joseph A. Insley, Yu-Hsiang Lan, et al. 2023. “Scaling Computational Fluid Dynamics: In Situ Visualization of NekRS Using SENSEI.” In Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis (SC-W 2023). doi:10.1145/3624062.3624159. 
    @inproceedings{MateevitsiEtAl2023,
  author = {Mateevitsi, Victor A. and Bode, Mathis and Ferrier, Nicola and Fischer, Paul and G{\"{o}}bbert, Jens Henrik and Insley, Joseph A. and Lan, Yu-Hsiang and Min, Misun and Papka, Michael E. and Patel, Saumil and Rizzi, Silvio and Windgassen, Jonathan},
  booktitle = {Workshops of The International Conference on High Performance Computing, Network,
    Storage, and Analysis (SC-W 2023)},
  title = {{Scaling Computational Fluid Dynamics: In Situ Visualization of NekRS using SENSEI}},
  year = {2023},
  doi = {10.1145/3624062.3624159}
}

Future plans

References

  1. Fischer, Paul, Stefan Kerkemeier, Misun Min, Yu-Hsiang Lan, Malachi Phillips, Thilina Rathnayake, Elia Merzari, et al. 2022. “NekRS, a GPU-Accelerated Spectral Element Navier–Stokes Solver.” Parallel Computing 114. Elsevier: 102982. 
    @article{FischerEtAl2022,
  title = {NekRS, a GPU-accelerated spectral element Navier--Stokes solver},
  author = {Fischer, Paul and Kerkemeier, Stefan and Min, Misun and Lan, Yu-Hsiang and Phillips, Malachi and Rathnayake, Thilina and Merzari, Elia and Tomboulides, Ananias and Karakus, Ali and Chalmers, Noel and Warburton, Tim},
  journal = {Parallel Computing},
  volume = {114},
  pages = {102982},
  year = {2022},
  publisher = {Elsevier}
}
  2. Ayachit, Utkarsh, Brad Whitlock, Matthew Wolf, Burlen Loring, Berk Geveci, David Lonie, and E Wes Bethel. 2016. “The SENSEI Generic in Situ Interface.” In 2016 Second Workshop on in Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV), 40–44. IEEE. 
    @inproceedings{AyachitEtAl2016,
  title = {The SENSEI generic in situ interface},
  author = {Ayachit, Utkarsh and Whitlock, Brad and Wolf, Matthew and Loring, Burlen and Geveci, Berk and Lonie, David and Bethel, E Wes},
  booktitle = {2016 second workshop on in situ infrastructures for enabling extreme-scale
      Analysis and visualization (ISAV)},
  pages = {40--44},
  year = {2016},
  organization = {IEEE}
}
  3. Bergman, Keren, Shekhar Borkar, Dan Campbell, William Carlson, William Dally, Monty Denneau, Paul Franzon, et al. 2008. “Exascale Computing Study: Technology Challenges in Achieving Exascale Systems.” Defense Advanced Research Projects Agency Information Processing Techniques Office (DARPA IPTO), Tech. Rep 15: 181. 
    @article{BergmanEtAl2008,
  title = {Exascale computing study: Technology challenges in achieving exascale systems},
  author = {Bergman, Keren and Borkar, Shekhar and Campbell, Dan and Carlson, William and Dally, William and Denneau, Monty and Franzon, Paul and Harrod, William and Hill, Kerry and Hiller, Jon and others},
  journal = {Defense Advanced Research Projects Agency Information Processing Techniques Office
      (DARPA IPTO), Tech. Rep},
  volume = {15},
  pages = {181},
  year = {2008}
}