American Institute of Mathematical Sciences

Advanced
doi: 10.3934/mcrf.2020027
Online First

Online First articles are published articles within a journal that have not yet been assigned to a formal issue. This means they do not yet have a volume number, issue number, or page numbers assigned to them, however, they can still be found and cited using their DOI (Digital Object Identifier). Online First publication benefits the research community by making new scientific discoveries known as quickly as possible.

Readers can access Online First articles via the “Online First” tab for the selected journal.

Stochastic optimal control — A concise introduction

Jiongmin Yong

Department of Mathematics, University of Central Florida, Orlando, FL 32816, USA

Received  August 2019 Revised  January 2020 Early access June 2020

Fund Project: This work is supported in part by NSF Grant DMS-1812921

This is a concise introduction to stochastic optimal control theory. We assume that the readers have basic knowledge of real analysis, functional analysis, elementary probability, ordinary differential equations and partial differential equations. We will present the following topics: (ⅰ) A brief presentation of relevant results on stochastic analysis; (ⅱ) Formulation of stochastic optimal control problems; (ⅲ) Variational method and Pontryagin's maximum principle, together with a brief introduction of backward stochastic differential equations; (ⅳ) Dynamic programming method and viscosity solutions to Hamilton-Jacobi-Bellman equation; (ⅴ) Linear-quadratic optimal control problems, including a careful discussion on open-loop optimal controls and closed-loop optimal strategies, linear forward-backward stochastic differential equations, and Riccati equations.

Keywords: Optimal stochastic control, variational method, dynamic programming, Pontryagin maximum principle, Hamilton-Jacobi-Bellman equation, Riccati equation.
Mathematics Subject Classification: Primary: 93E20, 49K45, 49L05, 49L20, 49L25, 49N10; Secondary: 35D40, 35F21, 35Q93, 60H10.
Citation: Jiongmin Yong. Stochastic optimal control — A concise introduction. Mathematical Control & Related Fields, doi: 10.3934/mcrf.2020027
References:
[1]

M. G. Crandall and P. L. Lions, Viscosity solutions of Hamilton-Jacobi equations, Trans. Amer. Math. Soc., 277 (1983), 1-42.  doi: 10.1090/S0002-9947-1983-0690039-8.  Google Scholar

[2] L. C. Evans and R. F. Gariepy, Measure Theory and Fine Properties of Functions, CRC Press, Boca Raton, FL, 1992.   Google Scholar
[3]

W. H. Fleming and H. M. Soner, Controlled Markov Processes and Viscosity Solutions, Springer-Verlag, New York, 1993.  Google Scholar

[4]

A. Gary, D. Greenhalgh, L. Hu, X. Mao and J. Pan, A stochastic differential equation SIS epidemic model, SIAM J. Appl. Math., 71 (2011), 876-902.  doi: 10.1137/10081856X.  Google Scholar

[5] S. He, J. Wang and J. Yan, Semimartingale Theory and Stochastic Calculus, Science Press and CRC Press, Beijing, 1992.   Google Scholar
[6]

I. Karatzas and S. Shreve, Brownian Motion and Stochastic Calculus, Springer-Verlag, New York, 1988, 47–127. doi: 10.1007/978-1-4684-0302-2_2.  Google Scholar

[7]

E. Pardoux and S. Peng, Adapted solution of backward stochastic differential equations, Systems Control Lett., 14 (1990), 55-61.  doi: 10.1016/0167-6911(90)90082-6.  Google Scholar

[8]

S. Peng, A general stochastic maximum principle for optimal control problems, SIAM J. Control Optim., 28 (1990), 966-979.  doi: 10.1137/0328054.  Google Scholar

[9]

J. Sun, X. Li and J. Yong, Open-loop and closed-loop solvabilities for stochastic linear quadratic optimal control problems, SIAM J. Control Optim., 54 (2016), 2274-2308.  doi: 10.1137/15M103532X.  Google Scholar

[10]

J. Sun and J. Yong, Linear quadratic stochastic differential games: Open-loop and closed-loop saddle points, SIAM J. Control Optim., 52 (2014), 4082-4121.  doi: 10.1137/140953642.  Google Scholar

[11]

E. Tornatore, S. M. Buccellato and P. Vetro, Stability of a stochastic SIR system, Physica A, 354 (2005), 111-126.  doi: 10.1016/j.physa.2005.02.057.  Google Scholar

[12]

J. Yong and X. Y. Zhou, Stochastic Controls: Hamiltonian Systems and HJB Equations, Springer-Verlag, New York, 1999. doi: 10.1007/978-1-4612-1466-3.  Google Scholar

show all references

References:
[1]

M. G. Crandall and P. L. Lions, Viscosity solutions of Hamilton-Jacobi equations, Trans. Amer. Math. Soc., 277 (1983), 1-42.  doi: 10.1090/S0002-9947-1983-0690039-8.  Google Scholar

[2] L. C. Evans and R. F. Gariepy, Measure Theory and Fine Properties of Functions, CRC Press, Boca Raton, FL, 1992.   Google Scholar
[3]

W. H. Fleming and H. M. Soner, Controlled Markov Processes and Viscosity Solutions, Springer-Verlag, New York, 1993.  Google Scholar

[4]

A. Gary, D. Greenhalgh, L. Hu, X. Mao and J. Pan, A stochastic differential equation SIS epidemic model, SIAM J. Appl. Math., 71 (2011), 876-902.  doi: 10.1137/10081856X.  Google Scholar

[5] S. He, J. Wang and J. Yan, Semimartingale Theory and Stochastic Calculus, Science Press and CRC Press, Beijing, 1992.   Google Scholar
[6]

I. Karatzas and S. Shreve, Brownian Motion and Stochastic Calculus, Springer-Verlag, New York, 1988, 47–127. doi: 10.1007/978-1-4684-0302-2_2.  Google Scholar

[7]

E. Pardoux and S. Peng, Adapted solution of backward stochastic differential equations, Systems Control Lett., 14 (1990), 55-61.  doi: 10.1016/0167-6911(90)90082-6.  Google Scholar

[8]

S. Peng, A general stochastic maximum principle for optimal control problems, SIAM J. Control Optim., 28 (1990), 966-979.  doi: 10.1137/0328054.  Google Scholar

[9]

J. Sun, X. Li and J. Yong, Open-loop and closed-loop solvabilities for stochastic linear quadratic optimal control problems, SIAM J. Control Optim., 54 (2016), 2274-2308.  doi: 10.1137/15M103532X.  Google Scholar

[10]

J. Sun and J. Yong, Linear quadratic stochastic differential games: Open-loop and closed-loop saddle points, SIAM J. Control Optim., 52 (2014), 4082-4121.  doi: 10.1137/140953642.  Google Scholar

[11]

E. Tornatore, S. M. Buccellato and P. Vetro, Stability of a stochastic SIR system, Physica A, 354 (2005), 111-126.  doi: 10.1016/j.physa.2005.02.057.  Google Scholar

[12]

J. Yong and X. Y. Zhou, Stochastic Controls: Hamiltonian Systems and HJB Equations, Springer-Verlag, New York, 1999. doi: 10.1007/978-1-4612-1466-3.  Google Scholar

[1]

Xuhui Wang, Lei Hu. A new method to solve the Hamilton-Jacobi-Bellman equation for a stochastic portfolio optimization model with boundary memory. Journal of Industrial & Management Optimization, 2021  doi: 10.3934/jimo.2021137
[2]

Steven Richardson, Song Wang. The viscosity approximation to the Hamilton-Jacobi-Bellman equation in optimal feedback control: Upper bounds for extended domains. Journal of Industrial & Management Optimization, 2010, 6 (1) : 161-175. doi: 10.3934/jimo.2010.6.161
[3]

Zhen-Zhen Tao, Bing Sun. A feedback design for numerical solution to optimal control problems based on Hamilton-Jacobi-Bellman equation. Electronic Research Archive, 2021, 29 (5) : 3429-3447. doi: 10.3934/era.2021046
[4]

Jean-Claude Zambrini. On the geometry of the Hamilton-Jacobi-Bellman equation. Journal of Geometric Mechanics, 2009, 1 (3) : 369-387. doi: 10.3934/jgm.2009.1.369
[5]

Bian-Xia Yang, Shanshan Gu, Guowei Dai. Existence and multiplicity for Hamilton-Jacobi-Bellman equation. Communications on Pure & Applied Analysis, 2021, 20 (11) : 3767-3793. doi: 10.3934/cpaa.2021130
[6]

Daniele Castorina, Annalisa Cesaroni, Luca Rossi. On a parabolic Hamilton-Jacobi-Bellman equation degenerating at the boundary. Communications on Pure & Applied Analysis, 2016, 15 (4) : 1251-1263. doi: 10.3934/cpaa.2016.15.1251
[7]

Guy Barles, Ariela Briani, Emmanuel Trélat. Value function for regional control problems via dynamic programming and Pontryagin maximum principle. Mathematical Control & Related Fields, 2018, 8 (3&4) : 509-533. doi: 10.3934/mcrf.2018021
[8]

Joan-Andreu Lázaro-Camí, Juan-Pablo Ortega. The stochastic Hamilton-Jacobi equation. Journal of Geometric Mechanics, 2009, 1 (3) : 295-315. doi: 10.3934/jgm.2009.1.295
[9]

Stefan Doboszczak, Manil T. Mohan, Sivaguru S. Sritharan. Pontryagin maximum principle for the optimal control of linearized compressible navier-stokes equations with state constraints. Evolution Equations & Control Theory, 2020  doi: 10.3934/eect.2020110
[10]

Nicolas Forcadel, Mamdouh Zaydan. A comparison principle for Hamilton-Jacobi equation with moving in time boundary. Evolution Equations & Control Theory, 2019, 8 (3) : 543-565. doi: 10.3934/eect.2019026
[11]

Mohamed Assellaou, Olivier Bokanowski, Hasnaa Zidani. Error estimates for second order Hamilton-Jacobi-Bellman equations. Approximation of probabilistic reachable sets. Discrete & Continuous Dynamical Systems, 2015, 35 (9) : 3933-3964. doi: 10.3934/dcds.2015.35.3933
[12]

Zhen Wu, Feng Zhang. Maximum principle for discrete-time stochastic optimal control problem and stochastic game. Mathematical Control & Related Fields, 2021  doi: 10.3934/mcrf.2021031
[13]

Tomoki Ohsawa, Anthony M. Bloch. Nonholonomic Hamilton-Jacobi equation and integrability. Journal of Geometric Mechanics, 2009, 1 (4) : 461-481. doi: 10.3934/jgm.2009.1.461
[14]

Nalini Anantharaman, Renato Iturriaga, Pablo Padilla, Héctor Sánchez-Morgado. Physical solutions of the Hamilton-Jacobi equation. Discrete & Continuous Dynamical Systems - B, 2005, 5 (3) : 513-528. doi: 10.3934/dcdsb.2005.5.513
[15]

María Barbero-Liñán, Manuel de León, David Martín de Diego, Juan C. Marrero, Miguel C. Muñoz-Lecanda. Kinematic reduction and the Hamilton-Jacobi equation. Journal of Geometric Mechanics, 2012, 4 (3) : 207-237. doi: 10.3934/jgm.2012.4.207
[16]

Larry M. Bates, Francesco Fassò, Nicola Sansonetto. The Hamilton-Jacobi equation, integrability, and nonholonomic systems. Journal of Geometric Mechanics, 2014, 6 (4) : 441-449. doi: 10.3934/jgm.2014.6.441
[17]

Xiao-Li Ding, Iván Area, Juan J. Nieto. Controlled singular evolution equations and Pontryagin type maximum principle with applications. Evolution Equations & Control Theory, 2021  doi: 10.3934/eect.2021059
[18]

Shaolin Ji, Xiaole Xue. A stochastic maximum principle for linear quadratic problem with nonconvex control domain. Mathematical Control & Related Fields, 2019, 9 (3) : 495-507. doi: 10.3934/mcrf.2019022
[19]

Yan Wang, Yanxiang Zhao, Lei Wang, Aimin Song, Yanping Ma. Stochastic maximum principle for partial information optimal investment and dividend problem of an insurer. Journal of Industrial & Management Optimization, 2018, 14 (2) : 653-671. doi: 10.3934/jimo.2017067
[20]

Shanjian Tang. A second-order maximum principle for singular optimal stochastic controls. Discrete & Continuous Dynamical Systems - B, 2010, 14 (4) : 1581-1599. doi: 10.3934/dcdsb.2010.14.1581

2020 Impact Factor: 1.284

Tools

Metrics

  • PDF downloads (1093)
  • HTML views (1241)
  • Cited by (0)

Other articles
by authors

[Back to Top]

Copyright © 2021 American Institute of Mathematical Sciences | Privacy Policy