American Institute of Mathematical Sciences

Advanced
doi: 10.3934/mcrf.2020027

Stochastic optimal control — A concise introduction

Jiongmin Yong

Department of Mathematics, University of Central Florida, Orlando, FL 32816, USA

Received  August 2019 Revised  January 2020 Published  June 2020

Fund Project: This work is supported in part by NSF Grant DMS-1812921

This is a concise introduction to stochastic optimal control theory. We assume that the readers have basic knowledge of real analysis, functional analysis, elementary probability, ordinary differential equations and partial differential equations. We will present the following topics: (ⅰ) A brief presentation of relevant results on stochastic analysis; (ⅱ) Formulation of stochastic optimal control problems; (ⅲ) Variational method and Pontryagin's maximum principle, together with a brief introduction of backward stochastic differential equations; (ⅳ) Dynamic programming method and viscosity solutions to Hamilton-Jacobi-Bellman equation; (ⅴ) Linear-quadratic optimal control problems, including a careful discussion on open-loop optimal controls and closed-loop optimal strategies, linear forward-backward stochastic differential equations, and Riccati equations.

Keywords: Optimal stochastic control, variational method, dynamic programming, Pontryagin maximum principle, Hamilton-Jacobi-Bellman equation, Riccati equation.
Mathematics Subject Classification: Primary: 93E20, 49K45, 49L05, 49L20, 49L25, 49N10; Secondary: 35D40, 35F21, 35Q93, 60H10.
Citation: Jiongmin Yong. Stochastic optimal control — A concise introduction. Mathematical Control & Related Fields, doi: 10.3934/mcrf.2020027
References:
[1]

M. G. Crandall and P. L. Lions, Viscosity solutions of Hamilton-Jacobi equations, Trans. Amer. Math. Soc., 277 (1983), 1-42.  doi: 10.1090/S0002-9947-1983-0690039-8.  Google Scholar

[2] L. C. Evans and R. F. Gariepy, Measure Theory and Fine Properties of Functions, CRC Press, Boca Raton, FL, 1992.   Google Scholar
[3]

W. H. Fleming and H. M. Soner, Controlled Markov Processes and Viscosity Solutions, Springer-Verlag, New York, 1993.  Google Scholar

[4]

A. Gary, D. Greenhalgh, L. Hu, X. Mao and J. Pan, A stochastic differential equation SIS epidemic model, SIAM J. Appl. Math., 71 (2011), 876-902.  doi: 10.1137/10081856X.  Google Scholar

[5] S. He, J. Wang and J. Yan, Semimartingale Theory and Stochastic Calculus, Science Press and CRC Press, Beijing, 1992.   Google Scholar
[6]

I. Karatzas and S. Shreve, Brownian Motion and Stochastic Calculus, Springer-Verlag, New York, 1988, 47–127. doi: 10.1007/978-1-4684-0302-2_2.  Google Scholar

[7]

E. Pardoux and S. Peng, Adapted solution of backward stochastic differential equations, Systems Control Lett., 14 (1990), 55-61.  doi: 10.1016/0167-6911(90)90082-6.  Google Scholar

[8]

S. Peng, A general stochastic maximum principle for optimal control problems, SIAM J. Control Optim., 28 (1990), 966-979.  doi: 10.1137/0328054.  Google Scholar

[9]

J. Sun, X. Li and J. Yong, Open-loop and closed-loop solvabilities for stochastic linear quadratic optimal control problems, SIAM J. Control Optim., 54 (2016), 2274-2308.  doi: 10.1137/15M103532X.  Google Scholar

[10]

J. Sun and J. Yong, Linear quadratic stochastic differential games: Open-loop and closed-loop saddle points, SIAM J. Control Optim., 52 (2014), 4082-4121.  doi: 10.1137/140953642.  Google Scholar

[11]

E. Tornatore, S. M. Buccellato and P. Vetro, Stability of a stochastic SIR system, Physica A, 354 (2005), 111-126.  doi: 10.1016/j.physa.2005.02.057.  Google Scholar

[12]

J. Yong and X. Y. Zhou, Stochastic Controls: Hamiltonian Systems and HJB Equations, Springer-Verlag, New York, 1999. doi: 10.1007/978-1-4612-1466-3.  Google Scholar

show all references

References:
[1]

M. G. Crandall and P. L. Lions, Viscosity solutions of Hamilton-Jacobi equations, Trans. Amer. Math. Soc., 277 (1983), 1-42.  doi: 10.1090/S0002-9947-1983-0690039-8.  Google Scholar

[2] L. C. Evans and R. F. Gariepy, Measure Theory and Fine Properties of Functions, CRC Press, Boca Raton, FL, 1992.   Google Scholar
[3]

W. H. Fleming and H. M. Soner, Controlled Markov Processes and Viscosity Solutions, Springer-Verlag, New York, 1993.  Google Scholar

[4]

A. Gary, D. Greenhalgh, L. Hu, X. Mao and J. Pan, A stochastic differential equation SIS epidemic model, SIAM J. Appl. Math., 71 (2011), 876-902.  doi: 10.1137/10081856X.  Google Scholar

[5] S. He, J. Wang and J. Yan, Semimartingale Theory and Stochastic Calculus, Science Press and CRC Press, Beijing, 1992.   Google Scholar
[6]

I. Karatzas and S. Shreve, Brownian Motion and Stochastic Calculus, Springer-Verlag, New York, 1988, 47–127. doi: 10.1007/978-1-4684-0302-2_2.  Google Scholar

[7]

E. Pardoux and S. Peng, Adapted solution of backward stochastic differential equations, Systems Control Lett., 14 (1990), 55-61.  doi: 10.1016/0167-6911(90)90082-6.  Google Scholar

[8]

S. Peng, A general stochastic maximum principle for optimal control problems, SIAM J. Control Optim., 28 (1990), 966-979.  doi: 10.1137/0328054.  Google Scholar

[9]

J. Sun, X. Li and J. Yong, Open-loop and closed-loop solvabilities for stochastic linear quadratic optimal control problems, SIAM J. Control Optim., 54 (2016), 2274-2308.  doi: 10.1137/15M103532X.  Google Scholar

[10]

J. Sun and J. Yong, Linear quadratic stochastic differential games: Open-loop and closed-loop saddle points, SIAM J. Control Optim., 52 (2014), 4082-4121.  doi: 10.1137/140953642.  Google Scholar

[11]

E. Tornatore, S. M. Buccellato and P. Vetro, Stability of a stochastic SIR system, Physica A, 354 (2005), 111-126.  doi: 10.1016/j.physa.2005.02.057.  Google Scholar

[12]

J. Yong and X. Y. Zhou, Stochastic Controls: Hamiltonian Systems and HJB Equations, Springer-Verlag, New York, 1999. doi: 10.1007/978-1-4612-1466-3.  Google Scholar

[1]

Diana Keller. Optimal control of a linear stochastic Schrödinger equation. Conference Publications, 2013, 2013 (special) : 437-446. doi: 10.3934/proc.2013.2013.437
[2]

Shanjian Tang, Fu Zhang. Path-dependent optimal stochastic control and viscosity solution of associated Bellman equations. Discrete & Continuous Dynamical Systems, 2015, 35 (11) : 5521-5553. doi: 10.3934/dcds.2015.35.5521
[3]

Shi'an Wang, N. U. Ahmed. Optimal control and stabilization of building maintenance units based on minimum principle. Journal of Industrial & Management Optimization, 2021, 17 (4) : 1713-1727. doi: 10.3934/jimo.2020041
[4]

Luke Finlay, Vladimir Gaitsgory, Ivan Lebedev. Linear programming solutions of periodic optimization problems: approximation of the optimal control. Journal of Industrial & Management Optimization, 2007, 3 (2) : 399-413. doi: 10.3934/jimo.2007.3.399
[5]

Simone Cacace, Maurizio Falcone. A dynamic domain decomposition for the eikonal-diffusion equation. Discrete & Continuous Dynamical Systems - S, 2016, 9 (1) : 109-123. doi: 10.3934/dcdss.2016.9.109
[6]

Livia Betz, Irwin Yousept. Optimal control of elliptic variational inequalities with bounded and unbounded operators. Mathematical Control & Related Fields, 2021  doi: 10.3934/mcrf.2021009
[7]

Vladimir Gaitsgory, Ilya Shvartsman. Linear programming estimates for Cesàro and Abel limits of optimal values in optimal control problems. Discrete & Continuous Dynamical Systems - B, 2021  doi: 10.3934/dcdsb.2021102
[8]

Fabio Camilli, Serikbolsyn Duisembay, Qing Tang. Approximation of an optimal control problem for the time-fractional Fokker-Planck equation. Journal of Dynamics & Games, 2021  doi: 10.3934/jdg.2021013
[9]

Alexandr Mikhaylov, Victor Mikhaylov. Dynamic inverse problem for Jacobi matrices. Inverse Problems & Imaging, 2019, 13 (3) : 431-447. doi: 10.3934/ipi.2019021
[10]

Shuting Chen, Zengji Du, Jiang Liu, Ke Wang. The dynamic properties of a generalized Kawahara equation with Kuramoto-Sivashinsky perturbation. Discrete & Continuous Dynamical Systems - B, 2021  doi: 10.3934/dcdsb.2021098
[11]

Mohammed Abdelghany, Amr B. Eltawil, Zakaria Yahia, Kazuhide Nakata. A hybrid variable neighbourhood search and dynamic programming approach for the nurse rostering problem. Journal of Industrial & Management Optimization, 2021, 17 (4) : 2051-2072. doi: 10.3934/jimo.2020058
[12]

Boris Kramer, John R. Singler. A POD projection method for large-scale algebraic Riccati equations. Numerical Algebra, Control & Optimization, 2016, 6 (4) : 413-435. doi: 10.3934/naco.2016018
[13]

Ahmad El Hajj, Hassan Ibrahim, Vivian Rizik. $ BV $ solution for a non-linear Hamilton-Jacobi system. Discrete & Continuous Dynamical Systems, 2021, 41 (7) : 3273-3293. doi: 10.3934/dcds.2020405
[14]

Olivier Ley, Erwin Topp, Miguel Yangari. Some results for the large time behavior of Hamilton-Jacobi equations with Caputo time derivative. Discrete & Continuous Dynamical Systems, 2021, 41 (8) : 3555-3577. doi: 10.3934/dcds.2021007
[15]

Woocheol Choi, Youngwoo Koh. On the splitting method for the nonlinear Schrödinger equation with initial data in $ H^1 $. Discrete & Continuous Dynamical Systems, 2021, 41 (8) : 3837-3867. doi: 10.3934/dcds.2021019
[16]

Amanda E. Diegel. A C0 interior penalty method for the Cahn-Hilliard equation. Electronic Research Archive, , () : -. doi: 10.3934/era.2021030
[17]

Mohamed Ouzahra. Approximate controllability of the semilinear reaction-diffusion equation governed by a multiplicative control. Discrete & Continuous Dynamical Systems - B, 2021  doi: 10.3934/dcdsb.2021081
[18]

Kehan Si, Zhenda Xu, Ka Fai Cedric Yiu, Xun Li. Open-loop solvability for mean-field stochastic linear quadratic optimal control problems of Markov regime-switching system. Journal of Industrial & Management Optimization, 2021  doi: 10.3934/jimo.2021074
[19]

Ardeshir Ahmadi, Hamed Davari-Ardakani. A multistage stochastic programming framework for cardinality constrained portfolio optimization. Numerical Algebra, Control & Optimization, 2017, 7 (3) : 359-377. doi: 10.3934/naco.2017023
[20]

Tobias Breiten, Sergey Dolgov, Martin Stoll. Solving differential Riccati equations: A nonlinear space-time method using tensor trains. Numerical Algebra, Control & Optimization, 2021, 11 (3) : 407-429. doi: 10.3934/naco.2020034

2019 Impact Factor: 0.857

Tools

Metrics

  • PDF downloads (456)
  • HTML views (558)
  • Cited by (0)

Other articles
by authors

[Back to Top]

Copyright © 2021 American Institute of Mathematical Sciences