Deep Neural Networks for CFD Simulations

Research topic and goals

Simulation-based optimization in engineering requires high computational power and may benefit from system heterogeneity. Highly scalable CFD solvers have been developed at R-CCS, JSC, and BSC to conduct large-scale simulations. R-CCS focuses on full-scale simulations of vehicles aerodynamics. JSC and BSC investigate respiratory flows to improve medical treatments. Due to massive amounts of data produced, the analysis of the results is becoming a time-consuming process. Data-driven approaches have the potential to speed up analyses or to even replace certain physics in simulations. Deep learning methods have proven to be a fast alternative to extract multi-scale features from high-dimensional data sets. Deep neural networks (DNN) are able to predict the steady-state flow and aerodynamic coefficients around bluff bodies and airfoils, and unsteady laminar vortex shedding over circular bodies. Autoencoders (AE) or generative models, like variational AEs or generative adversarial networks (GANs), have shown great potential in predicting 2D and 3D flow fields.

It is the aim of the proposed cooperation to tie in with these topics and to develop methods to efficiently compute and analyze complex optimization setups in engineering utilizing heterogeneous architectures. The joint project focuses on the following three activities:

• The first activity concentrates on machine learning (ML)-based reduced order models (ROM) for the reconstruction of flow fields. Conventional linear ROM approaches, e.g., proper orthogonal decomposition (POD), cannot successfully reproduce the the nonlinear behavior of high Reynolds number (Re) flow fields. To address this problem, recently, nonlinear ROMs based on neural networks are attracting attention and are studied within this project.
• The second activity involves physics-informed neural networks (PINNs) for the prediction of flow fields. PINNs integrate the physical laws based on the governing equations along with constraints given by initial and boundary conditions in the loss function of a DNN. In the current project, they are trained with sparse spatial and/or temporal data of a computational domain to predict flow fields of the complete domain. The developments will be used for predicting flow based on sparse data from experimental fluid dynamics, e.g., the sparse pressure and velocity distributions at the surface of a road vehicle.
• The third activity focuses on flow predictions with graph neural networks (GNNs). GNNs can be applied to any shape or volume represented by a graph, e.g., triangulated shapes, or computational grids. Convolutional filters in GNNs operate on nodes and their neighboring nodes. This allows a more efficient training compared to convolutional neural networks (CNNs), whose convolutional filters are rectangular and operate in Cartesian directions. In this project, GNNs are trained with varying geometries of a flow configuration to predict the flow fields in new geometries that have not been part of the training data. The networks are developed for two types of training strategies: (i) in a supervised data-driven manner with ground truth data coming from CFD simulations and (ii) in an unsupervised data-free manner, where the network is guided with physical loss functions towards physically correct flow fields.

Bilateral support activities will lead to knowledge exchange with respect to the different hardware available at the partners’ sites, in CFD methods, and in deep learning approaches. Hence, expertise of the centers in these fields are strongly promoted in the course of this project. To foster the cooperation, mutual short-time stays of the involved scientists are planned.

Results for 2019/2020

None.

Results for 2020/2021

Due to the CODID-19 pandemic, the activities in the project had to be reduced. Unfortunately, a planned research stay of JSC’s Ph.D. student Mario Rüttgers at R-CCS could not take place. An exchange of research ideas took, however, place online via video conferences (see Visits and meetings below).

JSC was able to continue its work on DNNs. The architecture for such a neural network was developed to predict the sound pressure level in a 2D square domain. It was trained by numerical results from up to 20,000 GPU-based lattice-Boltzmann simulations that include randomly distributed rectangular and circular objects, and monopole sources. Types of boundary conditions, the monopole locations, and cell distances for objects and monopoles serve as input to the network. Parameters were studied to tune the predictions and to increase their accuracy. An optimal choice of the parameters lead to network-predicted results that are in good agreement with the simulated findings. The results have been presented at the International Conference on High Performance Computing - ISC High Performance 2020. A peer-reviewed article has been published in the corresponding Springer series as part of the Lecture Notes in Computer Science (LNCS)(Rüttgers et al. 2020).

R-CCS contributed with their talk “Distributed Learning for Three-dimensional Flow Field Mode Decomposition on Fugaku” to the 34th Computational Fluid Dynamics Symposium, which was held online on Dec. 21-23, 2020 (K. Ando, K. Onishi, R. Bale, M. Tsubokura, A. Kuroda, and K. Minami). In their research, R-CCS developed and investigated non-linear CNN-based mode decompositioning methods for three-dimensional flow fields around circular cylinder configurations. For the distributed learning, up to 25,250 nodes (1,212,000 cores) on the supercomputer Fugaku were used. Scalable model parallelization techniques for auto-encoder neural networks with a sustained performance of 7.8PFLOPs on 25,250 nodes were implemented.

Results for 2021/2022

The still ongoing pandemic lead again to reduced project activities. Nevertheless, both groups from JSC and R-CCS were able to continue their research.

JSC mainly worked on developing and integrating novel AI technologies into their CFD workflows. Reinforcement learning (RL) was implemented for shape optimization problems. For a generic case, i.e., a 2D channel with a constriction, RL was used to satisfy an optimum criterion that is a function of the pressure loss in the channel and the increase of the temperature given by the heat transfer at the channel wall. The work was presented at the International Conference on High Performance Computing - ISC High Performance 2021. A peer-reviewed article was published in the corresponding Springer series as part of the Lecture Notes in Computer Science (LNCS)(Rüttgers et al. 2021). An application of this technique to the shape optimization of a human nasal cavity is currently ongoing. At the JLESC meeting in 02/2021, the results of automating CFD workflows with a focus on the application to nasal cavity simulation workflows was presented. These results were later integrated into a journal article (Rüttgers et al. 2022).

R-CCS continued their work on developing DNNs for mode decompositioning of flow fields. They also participated in the International Conference on High Performance Computing - ISC High Performance 2021, with a talk “Nonlinear Mode Decomposition and Reduced-Order Modeling for Three-Dimensional Cylinder Flow by Distributed Learning on Fugaku”. They used CNNs to perform a parallel mode decompositioning of the flow field and employed long short-term memory networks (LSTMs) to predict the temporal development of the flow via the modes. The performance of this method was evaluated for different numbers of modes. A peer-reviewed article was published in the corresponding Springer series as part of the Lecture Notes in Computer Science (LNCS)(Ando et al. 2021)

Results for 2022/2023

Rakesh Sarma and Rishabh Puri joined the JLESC activities from the JSC side, and Onishi Junya from the R-CCS side. JSC and R-CCS jointly developed PINNs for flow predictions in classical problems of fluid dynamics, i.e., Poisueille flow, potential flow, Blasius boundary layer flow, and the Taylor-Green vortex. Rhishabh Puri presented the results at the Platform for Advanced Scientific Computing 2023 (PASC23) conference in Davos, Switzerland.

JSC and R-CCS jointly developed a GNN for the prediction of flow around air foils. The GNN uses computational grids of the simulation framework “multiphysics-Aerodynamisches Institut Aachen” (m-AIA), an extended version of the formerly known “Zonal Flow Solver”(Lintermann, Meinke, and Schröder 2020), and automatically generates an adjacency matrix that contains information about nodes and neighbouring nodes. First steps have been initiated to develop an open access library for GNN-based flow predictions. The PINN and GNN activities were intensified by a research visit of Onishi Junya at JSC in August 2023.

Regarding the AI-based reduced-order model, in this fiscal year, the functionality of the model has been enhanced. That is, a variational autoencoder (VAE) has been integrated for outputting latent variables in the model. Due to this modification, the reduction performance was increased for turbulent flow simulation data. These results have been presented at the ParCFD2022 and WCCM2022 conferences.

Results for 2023/2024

Rishabh Puri starts his PhD programme at KIT, but continues the JLESC activities as an associated researcher. Hadrien Calmet joined the JLESC activities from the BSC side. After demonstrating the robustness of PINNs for flow predictions in classical flow problems, the activities regarding PINNs are extended to vehicle aerodynamics. That is, flow fields around a road vehicle are predicted based on sparse spatial data at the vehicles’s surface, to derive force coefficients such as drag or lift. In a first step, the sparse surface data are extracted from CFD simulations conducted by the R-CCS side, as discussed during a research visit of Mario Rüttgers at R-CCS in October 2023. After this proof-of-concept, the sparse surface data could stem from experimental measurements at a driving vehicle.

The activities regarding GNNs are extended to respiratory flows. Hadrien Calmet aims to investigate GNNs by initiating the first phase, which involves employing the existing 2D algorithm of Graph Convolutional Neural Networks (GCNN) (Ogoke et al. 2021) and evaluating its efficacy with a 3D dataset. The goal is to predict respiratory system flow features such as air resistance, wall shear stress, and energy flux within the human nasal cavity during inspiration. The initial step involves generating a virtual population through random scaling applied simultaneously to length, width, and height. Three distinct geometries are chosen, including an average one based on 35 healthy patients, a Caucasian healthy patient, and an Asian healthy patient. Once the neural network demonstrates satisfactory accuracy, a second phase is planned to assess the algorithm’s performance. This evaluation will utilize Paraver, a performance tool developed at BSC. Additionally, there is a consideration for the implementation of multi-GPU using PyTorch, allowing for a performance comparison. The final step involves testing the AI4HPC open-source library to train the dataset with the GNN algorithm and engaging in a discussion about the obtained results.

Regarding the AI-based reduced-order model, the scope of application is extended. First, the method is applied to the turbulent flow around a vehicle calculated with 260 million cells. Distributed training was conducted for over 14 hours using 55,000 Fugaku computing nodes. Since the model is still in the early stages of learning, small-period eddies could not yet be reproduced, but large-scale structure could be reconstructed. Second, the robustness of the model is addressed for a two-dimensional flow around two square cylinders, which have a varying distances. It is demonstrated that flow configurations that did not belong to the training data are successfully reproduced using 24 modes. It is planned to submit these results to the JLESC special issue of the journal “Future Generation Computer Systems”.

Results for 2024/2025

The work on PINNs in collaboration between JSC and RIKEN is published in the JLESC special issue of Future Generation Computer Systems(Puri et al. 2024). In this work, the accuracy and performance of PINNs is compared to DNNs for classical flow problems. The work quantifies and compares the accuracy of the networks when a fraction of training data is available for training. This comparison also considered the computational effort for training the networks. Furthermore, the effect of location of the training data in the computational domain and potential noise are also assessed. It is found that in most cases, PINNs outperform DNNs in terms of accuracy, especially when the training data is positioned in certain Regions of Interest, such as near the wall. These findings are especially interesting for future investigations in vehicle aerodynamics, where sensor data is sparse and mostly available close to the wall of the vehicle body.

Currently, RIKEN is leading another development to the PINN model, where it is extended to predict flow fields under arbitrary flow conditions, such as Reynolds numbers and inflow velocity distributions, by introducing a Deep Operator Network (DeepONet). This approach is expected to facilitate the performance of a large number of simulations for various scenarios at a low cost.

Hadrien Calmet from BSC is continuing his work on the GCNN applied to respiratory flow problems. The objective is to predict the pressure drop and wall shear stress for flow in the nasal cavity with GCNN. Additionally, a data augmentation strategy is developed to generate a virtual population from a limited amount of real data, addressing the problem of data accessibility and enabling generalization to unseen real geometries. The trainings are facilitated by the AI4HPC framework developed at JSC, which enables large scale trainings of ML models in HPC systems. The work is currenly under progress and is planned to be submitted to a journal.

In this project period, Mario Rüttgers from JSC started working at the Data-Driven Fluid Engineering (DDFE) Lab at Inha University in South Korea. However, he continues to be associated to the JLESC activities in this project. The current efforts are directed towards development of physics-aware GCNNs for different use-cases in respiratory flows, urban flows, and combustion. Apart from the baseline development, model parallelism is being investigated for the GCNN model with graph partitioning approaches in collaboration with JSC.

The work on AI-based reduced-order models between JSC and RIKEN is published in the JLESC special issue of Future Generation Computer Systems(Higashida et al. 2024). The robustness of an artificial neural network for model order reduction of flow field data is studied, using large-scale distributed learning on up to 6259 nodes of the Fugaku supercomputer. Flow around two square cylinders with varying center distances is analyzed using simulation data for training and testing. The network’s ability to reconstruct flow fields with 2, 12, and 24 modes is evaluated, showing that 2 modes fail to capture both low- and high-frequency structures, while 12 and 24 modes yield more accurate reconstructions, especially for high-frequency waves near the cylinders. With 24 modes, the network produces smooth velocity fields that reproduce all relevant flow features for all geometric variations. Comparisons with proper orthogonal decomposition (POD) using 24 modes reveal that the neural network achieves lower mean squared errors across all cases, highlighting its superiority over linear methods like POD for reconstructing non-linear flow fields.

Visits and meetings

• throughout 2020: discussions of hosting a JSC researcher at R-CCS (canceled due to the COVID-19 pandemic)
• 02.12.2020: R-CCS presents latest work on mode decomposition using ML for 3D cylinder cases to JSC.
• 23.03.2020: JSC presents previous work in the combined field of CFD, AI to the JLESC partners at R-CCS (application of AI methods to automatize CFD workflows, ML-based segmentation of computer tomography images in preparation of CFD computations of respiratory flows, prediction of sound pressure level distributions with DNNs).
• 08.12.2020 - 11.12.2020: Meeting of the PIs Makoto Tsubokura, Andreas Lintermann, and the researchers Keiji Onishi (R-CCS) and Mario Rüttgers in the framework of the COMPSAFE 2020 online conference.
• 24.02.2021 - 26.02.2021: Mario Rüttgers presents the jointly developed results at the 13th JLESC workshop, online.
• 04.07.2022 - 06.07.2022: Mario Rüttgers presents the jointly developed results at the Kobe Opportunities workshop organized by Makoto Tsubokura in Brussels, Belgium.
• 28.09.2022 - 30.09.2022: Mario Rüttgers presents the jointly developed results at the 14th JLESC workshop in Urbana-Champaign, USA.
• 21.03.2023 - 23.03.2023: Ando Kazuto presents the jointly developed results at the 15th JLESC workshop in Bordeaux, France.
• 03.07.2023 - 07.07.2023: Research exchange: Onishi Junya from R-CCS visits researchers from JSC at JSC.
• 24.10.2023 - 27.10.2023: Research exchange: Mario Rüttgers visits researchers from R-CCS at R-CCS.
• 16.04.2024 - 18.04.2024: Rakesh Sarma (JSC), Junya Onishi (R-CCS) and Hadrien Calmet (BSC) present the jointly developed results at the 16th JLESC workshop in Kobe, Japan.
• 02.09.2024 - 04.09.2024: Mario Rüttgers (JSC), Rishabh Puri (KIT), Rakesh Sarma (JSC), Hadrien Calmet (BSC) and Junya Onishi (R-CCS) presented at the 35th Parallel Computational Fluid Dynamics (ParCFD) conference in Bonn, Germany.
• 08.09.2024 - 11.09.2024: Makoto Tsubokura from R-CCS and Mario Rüttgers from JSC presented at the 15th International Conference on Parallel Processing and Applied Mathematics (PPAM), Ostrava, Czech Republic.

Impact and publications

1. Puri2024

   BibTeX Abstract

   Puri, Rishabh, Junya Onishi, Mario Rüttgers, Rakesh Sarma, Makoto Tsubokura, and Andreas Lintermann. 2024. “On the Choice of Physical Constraints in Artificial Neural Networks for Predicting Flow Fields.” Future Generation Computer Systems 161 (December): 361–75. https://doi.org/10.1016/j.future.2024.07.009.

   @article{Puri2024,
     author = {Puri, Rishabh and Onishi, Junya and R{\"{u}}ttgers, Mario and Sarma, Rakesh and Tsubokura, Makoto and Lintermann, Andreas},
     doi = {10.1016/j.future.2024.07.009},
     issn = {0167739X},
     journal = {Future Generation Computer Systems},
     month = dec,
     pages = {361--375},
     title = {{On the choice of physical constraints in artificial neural networks for predicting flow fields}},
     url = {https://linkinghub.elsevier.com/retrieve/pii/S0167739X24003728},
     volume = {161},
     year = {2024}
   }
   

2. Higashida2024

   BibTeX Abstract

   Higashida, Aito, Kazuto Ando, Mario Rüttgers, Andreas Lintermann, and Makoto Tsubokura. 2024. “Robustness Evaluation of Large-Scale Machine Learning-Based Reduced Order Models for Reproducing Flow Fields.” Future Generation Computer Systems 159 (October): 243–54. https://doi.org/10.1016/j.future.2024.05.005.

   @article{Higashida2024,
     author = {Higashida, Aito and Ando, Kazuto and R{\"{u}}ttgers, Mario and Lintermann, Andreas and Tsubokura, Makoto},
     doi = {10.1016/j.future.2024.05.005},
     issn = {0167739X},
     journal = {Future Generation Computer Systems},
     month = oct,
     pages = {243--254},
     title = {{Robustness evaluation of large-scale machine learning-based reduced order models for reproducing flow fields}},
     url = {https://linkinghub.elsevier.com/retrieve/pii/S0167739X24002176},
     volume = {159},
     year = {2024}
   }
   

   The robustness of an artificial neural network that performs model order reduction for flow field data is studied. The network is trained with a large-scale distributed learning approach using up to 6, 259 nodes of the supercomputer Fugaku. Flow around two square cylinders with a varying distance between their centers is investigated. The network is trained and tested with data from numerical simulations. First, the capability to reproduce flow fields with 2, 12, and 24 modes is investigated by comparing the reconstructed flow data to simulated data. It is shown, that reconstructions based on 2 modes cannot capture both, low- and high-frequency flow structures correctly, whereas predictions based on 12 and 24 modes yield improved flow fields, especially in the case of high-frequency waves in the vicinity of the square cylinders. Reconstructions with 24 modes provide smooth velocity fields that reproduce all relevant low- and high-frequency waves for all variations of the distance between the two square cylinders. Second, the performance of the machine learning-based reconstructions are compared to proper orthogonal decomposition, which is a commonly used reduced order model technique. The comparison only includes flow fields based on 24 modes. For all geometric variations, the mean squared errors of the reconstructions by the conventional method are higher than those of the machine learning model. This underlines the advantage of artificial neural networks over linear methods like proper orthogonal decomposition for tasks like reconstructing flow fields that are characterized by non-linear governing equations.
3. Ruttgers2022APIN

   BibTeX Abstract

   Rüttgers, Mario, Moritz Waldmann, Wolfgang Schröder, and Andreas Lintermann. 2022. “A Machine-Learning-Based Method for Automatizing Lattice-Boltzmann Simulations of Respiratory Flows.” Applied Intelligence, no. first online (January). https://doi.org/10.1007/s10489-021-02808-2.

   @article{Ruttgers2022APIN,
     author = {R{\"{u}}ttgers, Mario and Waldmann, Moritz and Schr{\"{o}}der, Wolfgang and Lintermann, Andreas},
     doi = {10.1007/s10489-021-02808-2},
     issn = {0924-669X},
     journal = {Applied Intelligence},
     month = jan,
     number = {first online},
     title = {{A machine-learning-based method for automatizing lattice-Boltzmann simulations of respiratory flows}},
     url = {https://link.springer.com/10.1007/s10489-021-02808-2},
     year = {2022}
   }
   

4. Ogoke2021

   BibTeX Abstract

   Ogoke, Francis, Kazem Meidani, Amirreza Hashemi, and Amir Barati Farimani. 2021. “Graph Convolutional Networks Applied to Unstructured Flow Field Data.” Machine Learning: Science and Technology 2 (September). https://doi.org/10.1088/2632-2153/ac1fc9.

   @article{Ogoke2021,
     author = {Ogoke, Francis and Meidani, Kazem and Hashemi, Amirreza and Barati Farimani, Amir},
     year = {2021},
     month = sep,
     pages = {},
     title = {Graph Convolutional Networks applied to unstructured flow field data},
     volume = {2},
     journal = {Machine Learning: Science and Technology},
     doi = {10.1088/2632-2153/ac1fc9}
   }
   

5. Ruttgers2021ISC

   BibTeX Abstract

   Rüttgers, Mario, Moritz Waldmann, Wolfgang Schröder, and Andreas Lintermann. 2021. “Machine-Learning-Based Control of Perturbed and Heated Channel Flows.” In High Performance Computing, Proceedings of the 36th International Conference, ISC High Performance 2021, 7–22. Frankfurt/Main, Germany: Springer International Publishing. https://doi.org/10.1007/978-3-030-90539-2_1.

   @incollection{Ruttgers2021ISC,
     address = {Frankfurt/Main, Germany},
     author = {R{\"{u}}ttgers, Mario and Waldmann, Moritz and Schr{\"{o}}der, Wolfgang and Lintermann, Andreas},
     booktitle = {High Performance Computing, Proceedings of the 36th International Conference, ISC High Performance 2021},
     doi = {10.1007/978-3-030-90539-2_1},
     pages = {7--22},
     publisher = {Springer International Publishing},
     title = {{Machine-Learning-Based Control of Perturbed and Heated Channel Flows}},
     url = {https://link.springer.com/10.1007/978-3-030-90539-2{\_}1},
     year = {2021}
   }
   

6. Ando2021ISC

   BibTeX Abstract

   Ando, Kazuto, Keiji Onishi, Rahul Bale, Makoto Tsubokura, Akiyoshi Kuroda, and Kazuo Minami. 2021. “Nonlinear Mode Decomposition and Reduced-Order Modeling for Three-Dimensional Cylinder Flow by Distributed Learning on Fugaku.” In High Performance Computing, Proceedings of the 36th International Conference, ISC High Performance 2021, 122–37. Frankfurt/Main, Germany: Springer International Publishing. https://doi.org/10.1007/978-3-030-90539-2_8.

   @incollection{Ando2021ISC,
     address = {Frankfurt/Main, Germany},
     author = {Ando, Kazuto and Onishi, Keiji and Bale, Rahul and Tsubokura, Makoto and Kuroda, Akiyoshi and Minami, Kazuo},
     booktitle = {High Performance Computing, Proceedings of the 36th International Conference, ISC High Performance 2021},
     doi = {10.1007/978-3-030-90539-2_8},
     pages = {122--137},
     publisher = {Springer International Publishing},
     title = {{Nonlinear Mode Decomposition and Reduced-Order Modeling for Three-Dimensional Cylinder Flow by Distributed Learning on Fugaku}},
     url = {https://link.springer.com/10.1007/978-3-030-90539-2{\_}8},
     year = {2021}
   }
   

7. Lintermann2020

   BibTeX Abstract

   Lintermann, Andreas, Matthias Meinke, and Wolfgang Schröder. 2020. “Zonal Flow Solver (ZFS): a Highly Efficient Multi- Physics Simulation Framework.” International Journal of Computational Fluid Dynamics 34 (March). https://doi.org/10.1080/10618562.2020.1742328.

   @article{Lintermann2020,
     author = {Lintermann, Andreas and Meinke, Matthias and Schröder, Wolfgang},
     year = {2020},
     month = mar,
     pages = {},
     title = {Zonal Flow Solver (ZFS): a highly efficient multi- physics simulation framework},
     volume = {34},
     journal = {International Journal of Computational Fluid Dynamics},
     doi = {10.1080/10618562.2020.1742328}
   }
   

8. Ruttgers2020

   BibTeX Abstract

   Rüttgers, Mario, Seong-Ryong Koh, Jenia Jitsev, Wolfgang Schröder, and Andreas Lintermann. 2020. “Prediction of Acoustic Fields Using a Lattice-Boltzmann Method and Deep Learning.” In High Performance Computing, Proceedings of the 35th International Conference, ISC High Performance 2020, 81–101. Frankfurt/Main, Germany: Springer International Publishing. https://doi.org/10.1007/978-3-030-59851-8_6.

   @incollection{Ruttgers2020,
     address = {Frankfurt/Main, Germany},
     author = {R{\"{u}}ttgers, Mario and Koh, Seong-Ryong and Jitsev, Jenia and Schr{\"{o}}der, Wolfgang and Lintermann, Andreas},
     booktitle = {High Performance Computing, Proceedings of the 35th International Conference, ISC High Performance 2020},
     doi = {10.1007/978-3-030-59851-8_6},
     pages = {81--101},
     publisher = {Springer International Publishing},
     title = {{Prediction of Acoustic Fields Using a Lattice-Boltzmann Method and Deep Learning}},
     url = {http://link.springer.com/10.1007/978-3-030-59851-8{\_}6},
     year = {2020}
   }
   

Future plans

Future plans are discussed in regular meetings. Furthermore, Makoto Tsubokura intends to visit JSC in 03/2025 to further discuss the future project plan.

References