Lagrangian dual framework for conservative neural network solutions of kinetic equations

Hyung Ju Hwang; Hwijae Son

doi:10.3934/krm.2021046

Article Contents

2022, Volume 15, Issue 4: 551-568. Doi: 10.3934/krm.2021046

This issue Previous Article Pointwise bounds for the Green's function for the Neumann-Laplace operator in

$ \text{R}^3 $

Next Article Magnetic confinement for the 2D axisymmetric relativistic Vlasov-Maxwell system in an annulus

Lagrangian dual framework for conservative neural network solutions of kinetic equations

Hyung Ju Hwang^1, , and
Hwijae Son^2,

1.
Department of Mathematics, Pohang University of Science and Technology, Pohang, Republic of Korea
2.
Stochastic Analysis and Application Research Center, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea

^* Corresponding author: Hyung Ju Hwang
^* Corresponding author: Hyung Ju Hwang

Received: May 31, 2021

Published: August 2022

Abstract Full Text(HTML) Figure(10) Related Papers Cited by

Abstract

In this paper, we propose a novel conservative formulation for solving kinetic equations via neural networks. More precisely, we formulate the learning problem as a constrained optimization problem with constraints that represent the physical conservation laws. The constraints are relaxed toward the residual loss function by the Lagrangian duality. By imposing physical conservation properties of the solution as constraints of the learning problem, we demonstrate far more accurate approximations of the solutions in terms of errors and the conservation laws, for the kinetic Fokker-Planck equation and the homogeneous Boltzmann equation.

Keywords:

Mathematics Subject Classification: Primary: 68T07, 82B40.

Citation:

Full Text(HTML)

Figure 1. Left: Relaxation to Maxwellian of f(t, x = 1, v) from an initial condition $ f_0^{(1)} $. Right: Relaxation to Maxwellian of f(t, x = 1, v) from an initial condition $ f_0^{(2)} $

Download: Full-size image PowerPoint slide

Figure 2. Numerical solutions of Test 1 at $ t = 0, \frac{1}{2}, $ and $ 1 $

Download: Full-size image PowerPoint slide

Figure 3. Left: Loss value in training epoch at log scale. Right: Actual error in training epoch at log scale

Download: Full-size image PowerPoint slide

Figure 4. Left: Time averaged mass in training epoch. Right: Total mass in time after training is finished

Download: Full-size image PowerPoint slide

Figure 5. Numerical solutions of Test 1 at $ t = 0, \frac{3}{2}, $ and $ 3 $

Download: Full-size image PowerPoint slide

Figure 6. Left: Loss value in training epoch at log scale. Right: Actual error in training epoch at log scale

Download: Full-size image PowerPoint slide

Figure 7. Left: Time averaged mass in training epoch. Right: Total mass in time after training is finished

Download: Full-size image PowerPoint slide

Figure 8. Left: Value of the loss $ \widehat{Loss}_B $ in training epoch. Right: $ L^{\infty}((0,1);L^2_{v_x,v_y}) $ error in training epoch

Download: Full-size image PowerPoint slide

Figure 9. Left: Total mass in time after the training is finished. Right: Kinetic energy in time after the training is finished

Download: Full-size image PowerPoint slide

Figure 10. Momentum in time after the training is finished

Download: Full-size image PowerPoint slide

Related Papers

Cited by

References

[1]	V. V. Aristov, Direct Methods for Solving the Boltzmann Equation and Study of Nonequilibrium Flows, Fluid Mechanics and its Applications, 60, Kluwer Academic Publishers Group, Dordrecht, 2001. doi: 10.1007/978-94-010-0866-2.
[2]	J. Berg and K. Nyström, A unified deep artificial neural network approach to partial differential equations in complex geometries, Neurocomputing, 317 (2018), 28-41. doi: 10.1016/j.neucom.2018.06.056.
[3]	D. P. Bertsekas, Multiplier methods: A survey, Automatica J. IFAC, 12 (1976), 133-145. doi: 10.1016/0005-1098(76)90077-7.
[4]	A. V. Bobylëv, Exact solutions of the Boltzmann equation, Dokl. Akad. Nauk SSSR, 225 (1975), 1296-1299.
[5]	L. L. Bonilla, J. A. Carrillo and J. Soler, Asymptotic behavior of an initial-boundary value problem for the Vlasov–Poisson–Fokker–Planck system, SIAM J. Appl. Math., 57 (1997), 1343-1372. doi: 10.1137/S0036139995291544.
[6]	G. Cybenko, Approximation by superpositions of a sigmoidal function, Math. Control Signals Systems, 2 (1989), 303-314. doi: 10.1007/BF02551274.
[7]	G. Dimarco, R. Loubère, J. Narski and T. Rey, An efficient numerical method for solving the Boltzmann equation in multidimensions, J. Comput. Phys., 353 (2018), 46-81. doi: 10.1016/j.jcp.2017.10.010.
[8]	G. Dimarco and L. Pareschi, Numerical methods for kinetic equations, Acta Numer., 23 (2014), 369-520. doi: 10.1017/S0962492914000063.
[9]	W. E, J. Han and A. Jentzen, Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations, Commun. Math. Stat., 5 (2017), 349-380. doi: 10.1007/s40304-017-0117-6.
[10]	W. E and B. Yu, The deep Ritz method: A deep learning-based numerical algorithm for solving variational problems, Commun. Math. Stat., 6 (2018), 1-12. doi: 10.1007/s40304-018-0127-z.
[11]	F. Filbet and G. Russo, Accurate numerical methods for the Boltzmann equation, in Modeling and Computational Methods for Kinetic Equations, Model. Simul. Sci. Eng. Technol., Birkhäuser Boston, Boston, MA, 2004,117–145.
[12]	F. Fioretto, P. Van Hentenryck, T. W. K. Mak, C. Tran, F. Baldo and M. Lombardi, Lagrangian duality for constrained deep learning, in Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track, Lecture Notes in Computer Science, 12461, Springer, Cham, 118–135. doi: 10.1007/978-3-030-67670-4_8.
[13]	J. Han, A. Jentzen and W. E, Solving high-dimensional partial differential equations using deep learning, Proc. Natl. Acad. Sci. USA, 115 (2018), 8505-8510. doi: 10.1073/pnas.1718942115.
[14]	K. Hornik, M. Stinchcombe and H. White, Multilayer feedforward networks are universal approximators, Neural Networks, 2 (1989), 359-366. doi: 10.1016/0893-6080(89)90020-8.
[15]	H. J. Hwang, J. W. Jang, H. Jo and J. Y. Lee, Trend to equilibrium for the kinetic Fokker-Planck equation via the neural network approach, J. Comput. Phys., 419 (2020), 25pp. doi: 10.1016/j.jcp.2020.109665.
[16]	H. Jo, H. Son, H. J. Hwang and E. H. Kim, Deep neural network approach to forward-inverse problems, Netw. Heterog. Media, 15 (2020), 247-259. doi: 10.3934/nhm.2020011.
[17]	E. Kharazmi, Z. Zhang and G. E. M. Karniadakis, hp-VPINNs: Variational physics-informed neural networks with domain decomposition, Comput. Methods Appl. Mech. Engrg., 374 (2021), 25pp. doi: 10.1016/j.cma.2020.113547.
[18]	D. P. Kingma and J. L. Ba, Adam: A method for stochastic optimization, preprint, arXiv: 1412.6980.
[19]	M. Krook and T. T. Wu, Exact solutions of the Boltzmann equation, Phys. Fluids, 20 (1977), 1589-1595. doi: 10.1063/1.861780.
[20]	I. E. Lagaris, A. Likas and D. I. Fotiadis, Artificial neural networks for solving ordinary and partial differential equations, IEEE Trans. Neural Networks, 9 (1998), 987-1000. doi: 10.1109/72.712178.
[21]	I. E. Lagaris, A. C. Likas and D. G. Papageorgiou, Neural-network methods for boundary value problems with irregular boundaries, IEEE Trans. Neural Networks, 11 (2000), 1041-1049. doi: 10.1109/72.870037.
[22]	J. Y. Lee, J. W. Jang and H. J. Hwang, The model reduction of the Vlasov-Poisson-Fokker-Planck system to the Poisson-Nernst-Planck system via the deep neural network approach, ESAIM Math. Model. Numer. Anal., 55 (2021), 1803-1846. doi: 10.1051/m2an/2021038.
[23]	M. Leshno, V. Y. Lin, A. Pinkus and S. Schocken, Multilayer feedforward networks with a nonpolynomial activation function can approximate any function, Neural Networks, 6 (1993), 861-867. doi: 10.1016/S0893-6080(05)80131-5.
[24]	X. Li, Simultaneous approximations of multivariate functions and their derivatives by neural networks with one hidden layer, Neurocomputing, 12 (1996), 327-343. doi: 10.1016/0925-2312(95)00070-4.
[25]	Y. Liao and P. Ming, Deep Nitsche method: Deep Ritz method with essential boundary conditions, Commun. Comput. Phys., 29 (2021), 1365-1384. doi: 10.4208/cicp.OA-2020-0219.
[26]	Q. Lou, X. Meng and G. E. Karniadakis, Physics-informed neural networks for solving forward and inverse flow problems via the Boltzmann-BGK formulation, J. Comput. Phys., 447 (2021), 20pp. doi: 10.1016/j.jcp.2021.110676.
[27]	L. Lu, X. Meng, Z. Mao and G. E. Karniadakis, DeepXDE: A deep learning library for solving differential equations, SIAM Rev., 63 (2021), 208-228. doi: 10.1137/19M1274067.
[28]	D. G. Luenberger, Introduction to Linear and Nonlinear Programming, Vol. 28, Addison-Wesley, Reading, MA, 1973.
[29]	L. Lyu, K. Wu, R. Du and J. Chen, Enforcing exact boundary and initial conditions in the deep mixed residual method, CSIAM Trans. Appl. Math., 2 (2021), 748-775. doi: 10.4208/csiam-am.SO-2021-0011.
[30]	P. Márquez-Neila, M. Salzmann and P. Fua, Imposing hard constraints on deep networks: Promises and limitations, preprint, arXiv: 1706.02025.
[31]	L. D. McClenny and U. Braga-Neto, Self-adaptive physics-informed neural networks using a soft attention mechanism, preprint, arXiv: 2009.04544.
[32]	J. Müller and M. Zeinhofer, Deep Ritz revisited, preprint, arXiv: 1912.03937.
[33]	J. Müller and M. Zeinhofer, Notes on exact boundary values in residual minimisation, preprint, arXiv: 2105.02550.
[34]	Y. Nandwani, A. Pathak and P. Singla, A primal dual formulation for deep learning with constraints., Available from: https://proceedings.neurips.cc/paper/2019/file/cf708fc1decf0337aded484f8f4519ae-Paper.pdf.
[35]	L. Pareschi and G. Russo, Numerical solution of the Boltzmann equation. I. Spectrally accurate approximation of the collision operator, SIAM J. Numer. Anal., 37 (2000), 1217-1245. doi: 10.1137/S0036142998343300.
[36]	A. Paszke, S. Gross, F. Massa, A. Lerer and J. Bradbury, et al., PyTorch: An imperative style, high-performance deep learning library, in Advances in Neural Information Processing Systems, 2019, 8024–8035. Available from: https://proceedings.neurips.cc/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf.
[37]	M. Raissi, P. Perdikaris and G. E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys., 378 (2019), 686-707. doi: 10.1016/j.jcp.2018.10.045.
[38]	S. N. Ravi, T. Dinh, V. S. Lokhande and V. Singh, Explicitly imposing constraints in deep networks via conditional gradients gives improved generalization and faster convergence, Proceedings of the AAAI Conference on Artificial Intelligence, 33 (2019), 4772-4779. doi: 10.1609/aaai.v33i01.33014772.
[39]	S. Sangalli, E. Erdil, A. Hoetker, O. Donati and E. Konukoglu, Constrained optimization to train neural networks on critical and under-represented classes, preprint, arXiv: 2102.12894.
[40]	J. Sirignano and K. Spiliopoulos, DGM: A deep learning algorithm for solving partial differential equations, J. Comput. Phys., 375 (2018), 1339-1364. doi: 10.1016/j.jcp.2018.08.029.
[41]	H. Son, J. W. Jang, W. J. Han and H. J. Hwang, Sobolev training for physics informed neural networks, preprint, arXiv: 2101.08932.
[42]	R. van der Meer, C. W. Oosterlee and A. Borovykh, Optimally weighted loss functions for solving PDEs with Neural Networks, J. Comput. Appl. Math., 405 (2022). doi: 10.1016/j.cam.2021.113887.
[43]	S. Wang, X. Yu and P. Perdikaris, When and why PINNs fail to train: A neural tangent kernel perspective, J. Comput. Phys., 449 (2022). doi: 10.1016/j.jcp.2021.110768.
[44]	S. Wollman and E. Ozizmir, A deterministic particle method for the Vlasov–Fokker–Planck equation in one dimension, J. Comput. Appl. Math., 213 (2008), 316-365. doi: 10.1016/j.cam.2007.01.008.