PINNsNTK
When and why PINNs fail to train: a neural tangent kernel perspective. Physics-informed neural networks (PINNs) have lately received great attention thanks to their flexibility in tackling a wide range of forward and inverse problems involving partial differential equations. However, despite their noticeable empirical success, little is known about how such constrained neural networks behave during their training via gradient descent. More importantly, even less is known about why such models sometimes fail to train at all. In this work, we aim to investigate these questions through the lens of the Neural Tangent Kernel (NTK); a kernel that captures the behavior of fully-connected neural networks in the infinite width limit during training via gradient descent. Specifically, we derive the NTK of PINNs and prove that, under appropriate conditions, it converges to a deterministic kernel that stays constant during training in the infinite-width limit. This allows us to analyze the training dynamics of PINNs through the lens of their limiting NTK and find a remarkable discrepancy in the convergence rate of the different loss components contributing to the total training error. To address this fundamental pathology, we propose a novel gradient descent algorithm that utilizes the eigenvalues of the NTK to adaptively calibrate the convergence rate of the total training error. Finally, we perform a series of numerical experiments to verify the correctness of our theory and the practical effectiveness of the proposed algorithms. The data and code accompanying this manuscript are publicly available at url{https://github.com/PredictiveIntelligenceLab/PINNsNTK}.
Keywords for this software
References in zbMATH (referenced in 25 articles , 1 standard article )
Showing results 1 to 20 of 25.
Sorted by year (- Bai, Jinshuai; Zhou, Ying; Ma, Yuwei; Jeong, Hyogu; Zhan, Haifei; Rathnayaka, Charith; Sauret, Emilie; Gu, Yuantong: A general neural particle method for hydrodynamics modeling (2022)
- Bihlo, Alex; Popovych, Roman O.: Physics-informed neural networks for the shallow-water equations on the sphere (2022)
- Chang, Zhipeng; Li, Ke; Zou, Xiufen; Xiang, Xueshuang: High order deep neural network for solving high frequency partial differential equations (2022)
- Chiu, Pao-Hsiung; Wong, Jian Cheng; Ooi, Chinchun; Dao, My Ha; Ong, Yew-Soon: CAN-PINN: a fast physics-informed neural network based on coupled-automatic-numerical differentiation method (2022)
- Cui, Tao; Wang, Ziming; Xiang, Xueshuang: An efficient neural network method with plane wave activation functions for solving Helmholtz equation (2022)
- Dong, Suchuan; Yang, Jielin: On computing the hyperparameter of extreme learning machines: algorithm and application to computational PDEs, and comparison with classical and high-order finite elements (2022)
- Gao, Yihang; Ng, Michael K.: Wasserstein generative adversarial uncertainty quantification in physics-informed neural networks (2022)
- Haghighat, Ehsan; Amini, Danial; Juanes, Ruben: Physics-informed neural network simulation of multiphase poroelasticity using stress-split sequential training (2022)
- Henkes, Alexander; Wessels, Henning; Mahnken, Rolf: Physics informed neural networks for continuum micromechanics (2022)
- Li, Jing; Tartakovsky, Alexandre M.: Physics-informed Karhunen-Loéve and neural network approximations for solving inverse differential equation problems (2022)
- Raynaud, Gaétan; Houde, Sébastien; Gosselin, Frédérick P.: ModalPINN: an extension of physics-informed neural networks with enforced truncated Fourier decomposition for periodic flow reconstruction using a limited number of imperfect sensors (2022)
- Rivera, Jon A.; Taylor, Jamie M.; Omella, Ángel J.; Pardo, David: On quadrature rules for solving partial differential equations using neural networks (2022)
- Trask, Nathaniel; Huang, Andy; Hu, Xiaozhe: Enforcing exact physics in scientific machine learning: a data-driven exterior calculus on graphs (2022)
- Wang, Hengjie; Planas, Robert; Chandramowlishwaran, Aparna; Bostanabad, Ramin: Mosaic flows: a transferable deep learning framework for solving PDEs on unseen domains (2022)
- Wang, Sifan; Wang, Hanwen; Perdikaris, Paris: Improved architectures and training algorithms for deep operator networks (2022)
- Wang, Sifan; Yu, Xinling; Perdikaris, Paris: When and why PINNs fail to train: a neural tangent kernel perspective (2022)
- Yuan, Lei; Ni, Yi-Qing; Deng, Xiang-Yun; Hao, Shuo: A-PINN: auxiliary physics informed neural networks for forward and inverse problems of nonlinear integro-differential equations (2022)
- Yu, Jeremy; Lu, Lu; Meng, Xuhui; Karniadakis, George Em: Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems (2022)
- Chen, Yifan; Hosseini, Bamdad; Owhadi, Houman; Stuart, Andrew M.: Solving and learning nonlinear PDEs with Gaussian processes (2021)
- Gu, Yiqi; Wang, Chunmei; Yang, Haizhao: Structure probing neural network deflation (2021)