Running Master's Thesis
Description
Scientific Machine Learning made significant progress in solving Partial Differential Equations (PDEs), shifting from traditional learning of a solution to a single, parameterized PDE towards learning the entire family of solutions. This shift towards learning the operator of the PDE has significantly improved the generalization capabilities of the models [1, 2]. In [3], the authors have shown that the generalization capabilities towards unseen PDEs can be improved by leveraging the idea of Transfer Learning, pre-training the model on a diverse set of PDEs and then fine-tuning the model on its downstream task.
We propose to improve generalizability by making such foundation models physics-aware by penalizing the model for violating the system's underlying physics, similar to the approach in PINNs. To achieve this, we introduce a loss term into the foundation model's training and transfer learning stages, characterized by the residuals of the various PDEs, capturing the violation of the system's underlying physics. Our goal is to investigate how the inclusion of this physics-based loss term affects the model's performance and its generalization capabilities on downstream tasks.
References
[1] Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Leaming Research, 24(89):1-97, 2023.
[2] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations, 2021.
[3] Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael W Mahoney, and Amir Gholami. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 71242-71262. Curran Associates, Inc., 2023.
[4] M. Raissi, P. Perdikaris, and G.E. Kamiadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[5] Zhiping Mao, Ameya D. Jagtap, and George Em Karniadakis. Physics-informed neural networks for high-speed flows. Computer Methods in Applied Mechanics and Engineering, 360:112789, 2020.
[6] Shengze Cai, Zhicheng Wang, Sifan Wang, Paris Perdikaris, and George Em Karniadakis. Physics-Informed Neural Networks for Heat Transfer Problems. Journal of Heat Transfer, 143(6):060801, 04 2021.
[7] Zhiwei Fang and Justin Zhan. A physics-informed neural network framework for pdes on 3d surfaces: Time independent problems. IEEE Access, 8:26328-26335, 2020.
[8] Minglang Yin, Xiaoning Zheng, Jay D. Humphrey, and George Em Karniadakis. Non-invasive inference of thrombus material properties with physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 375:113603, 2021.
[9] Francisco Sahli Costabal, Yibo Yang, Paris Perdikaris, Daniel E. Hurtado, and Ellen Kuhl. Physics-informed neural networks for cardiac activation mapping. Frontiers in Physics, 8, 2020.
[10] Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. ACM / IMS J. Data Sci., 1(3), may 2024.