American Institute of Mathematical Sciences

Advanced
  • Previous Article
    Determining the waveguide conductivity in a hyperbolic equation from a single measurement on the lateral boundary
  • MCRF Home
  • This Issue
  • Next Article
    A sparse Markov chain approximation of LQ-type stochastic control problems
September  2016, 6(3): 391-406. doi: 10.3934/mcrf.2016008

On the convergence of the Sakawa-Shindo algorithm in stochastic control

J. Frédéric Bonnans 1, , Justina Gianatti 2, and  Francisco J. Silva 3,

1. 

INRIA-Saclay and Centre de Mathématiques Appliquées, Ecole Polytechnique and Laboratoire de Finance des Marchés d'Énergie, 91128 Palaiseau, France
2. 

CIFASIS - Centro Internacional Franco Argentino, de Ciencias de la Información y de Sistemas, CONICET - UNR - AMU, S2000EZP Rosario, Argentina
3. 

Institut de recherche XLIM-DMI, UMR-CNRS 7252, Faculté des sciences et techniques, Université de Limoges, 87060 Limoges, France

Received  May 2015 Revised  August 2015 Published  August 2016

We analyze an algorithm for solving stochastic control problems, based on Pontryagin's maximum principle, due to Sakawa and Shindo in the deterministic case and extended to the stochastic setting by Mazliak. We assume that either the volatility is an affine function of the state, or the dynamics are linear. We obtain a monotone decrease of the cost functions as well as, in the convex case, the fact that the sequence of controls is minimizing, and converges to an optimal solution if it is bounded. In a specific case we interpret the algorithm as the gradient plus projection method and obtain a linear convergence rate to the solution.
Keywords: Pontryagin's principle, first order algorithm., Stochastic control.
Mathematics Subject Classification: Primary: 93E20, 49M05; Secondary: 49K45, 93E25, 60H10, 60H3.
Citation: J. Frédéric Bonnans, Justina Gianatti, Francisco J. Silva. On the convergence of the Sakawa-Shindo algorithm in stochastic control. Mathematical Control and Related Fields, 2016, 6 (3) : 391-406. doi: 10.3934/mcrf.2016008
References:
[1]

J. Backhoff and F. J. Silva, Sensitivity results in stochastic optimal control: A Lagrangian perspective, ESAIM: COCV, to appear. doi: 10.1051/cocv/2015039.

[2]

A. Bensoussan, Lectures on Stochastic Control, Lectures notes in Maths. Vol. 972, Springer-Verlag, Berlin, 1982.

[3]

A. Bensoussan, Stochastic maximum principle for distributed parameter systems, J. Franklin Inst., 315 (1983), 387-406, doi: 10.1016/0016-0032(83)90059-5.

[4]

J.-M. Bismut, Linear quadratic optimal stochastic control with random coefficients, SIAM J. Control Optimization, 14 (1976), 419-444. doi: 10.1137/0314028.

[5]

J.-M. Bismut, Conjugate convex functions in optimal stochastic control, J. Math. Anal. Appl., 44 (1973), 384-404. doi: 10.1016/0022-247X(73)90066-8.

[6]

J.-M. Bismut, An introductory approach to duality in optimal stochastic control, SIAM Rev., 20 (1978), 62-78. doi: 10.1137/1020004.

[7]

J. F. Bonnans, On an algorithm for optimal control using Pontryagin's maximum principle, SIAM J. Control Optim., 24 (1986), 579-588. doi: 10.1137/0324034.

[8]

J. F. Bonnans and F. J. Silva, First and second order necessary conditions for stochastic optimal control problems, Appl. Math. Optim., 65 (2012), 403-439. doi: 10.1007/s00245-012-9162-4.

[9]

A. Cadenillas and I. Karatzas, The stochastic maximum principle for linear convex optimal control with random coefficients, SIAM J. Control Optim., 33 (1995), 590-624. doi: 10.1137/S0363012992240722.

[10]

A. Goldstein, Convex programming in Hilbert space, Bull. Amer. Math. Soc., 70 (1964), 709-710. doi: 10.1090/S0002-9904-1964-11178-2.

[11]

U. G. Haussmann, Some examples of optimal stochastic controls or: The stochastic maximum principle at work, SIAM Rev., 23 (1981), 292-307. doi: 10.1137/1023062.

[12]

H. J. Kushner, Necessary conditions for continuous parameter stochastic optimization problems, SIAM J. Control, 10 (1972), 550-565. doi: 10.1137/0310041.

[13]

H. J. Kushner, On the stochastic maximum principle: Fixed time of control, J. Math. Anal. Appl., 11 (1965), 78-92. doi: 10.1016/0022-247X(65)90070-3.

[14]

H. J. Kushner and F. C. Schweppe, A maximum principle for stochastic control systems, J. Math. Anal. Appl., 8 (1964), 287-302. doi: 10.1016/0022-247X(64)90070-8.

[15]

L. Mazliak, An algorithm for solving a stochastic control problem, Stochastic analysis and applications, 14 (1996), 513-533. doi: 10.1080/07362999608809455.

[16]

L. Mou and J. Yong, A variational formula for stochastic controls and some applications, Pure Appl. Math. Q., 3 (2007), 539-567. doi: 10.4310/PAMQ.2007.v3.n2.a7.

[17]

S. G. Peng, A general stochastic maximum principle for optimal control problems, SIAM J. Control Optim., 28 (1990), 966-979. doi: 10.1137/0328054.

[18]

L. Pontryagin, V. Boltyanskiĭ, R. Gamkrelidze and E. Mishchenko, The Mathematical Theory of Optimal Processes, Gordon & Breach Science Publishers, New York, 1986, Reprint of the 1962 English translation.

[19]

Y. Sakawa and Y. Shindo, On global convergence of an algorithm for optimal control, IEEE Trans. Automat. Control, 25 (1980), 1149-1153. doi: 10.1109/TAC.1980.1102517.

[20]

J. Yong and X. Zhou, Stochastic Controls, Hamiltonian Systems and HJB Equations, Springer-Verlag, New York, 1999. doi: 10.1007/978-1-4612-1466-3.

show all references

References:
[1]

J. Backhoff and F. J. Silva, Sensitivity results in stochastic optimal control: A Lagrangian perspective, ESAIM: COCV, to appear. doi: 10.1051/cocv/2015039.

[2]

A. Bensoussan, Lectures on Stochastic Control, Lectures notes in Maths. Vol. 972, Springer-Verlag, Berlin, 1982.

[3]

A. Bensoussan, Stochastic maximum principle for distributed parameter systems, J. Franklin Inst., 315 (1983), 387-406, doi: 10.1016/0016-0032(83)90059-5.

[4]

J.-M. Bismut, Linear quadratic optimal stochastic control with random coefficients, SIAM J. Control Optimization, 14 (1976), 419-444. doi: 10.1137/0314028.

[5]

J.-M. Bismut, Conjugate convex functions in optimal stochastic control, J. Math. Anal. Appl., 44 (1973), 384-404. doi: 10.1016/0022-247X(73)90066-8.

[6]

J.-M. Bismut, An introductory approach to duality in optimal stochastic control, SIAM Rev., 20 (1978), 62-78. doi: 10.1137/1020004.

[7]

J. F. Bonnans, On an algorithm for optimal control using Pontryagin's maximum principle, SIAM J. Control Optim., 24 (1986), 579-588. doi: 10.1137/0324034.

[8]

J. F. Bonnans and F. J. Silva, First and second order necessary conditions for stochastic optimal control problems, Appl. Math. Optim., 65 (2012), 403-439. doi: 10.1007/s00245-012-9162-4.

[9]

A. Cadenillas and I. Karatzas, The stochastic maximum principle for linear convex optimal control with random coefficients, SIAM J. Control Optim., 33 (1995), 590-624. doi: 10.1137/S0363012992240722.

[10]

A. Goldstein, Convex programming in Hilbert space, Bull. Amer. Math. Soc., 70 (1964), 709-710. doi: 10.1090/S0002-9904-1964-11178-2.

[11]

U. G. Haussmann, Some examples of optimal stochastic controls or: The stochastic maximum principle at work, SIAM Rev., 23 (1981), 292-307. doi: 10.1137/1023062.

[12]

H. J. Kushner, Necessary conditions for continuous parameter stochastic optimization problems, SIAM J. Control, 10 (1972), 550-565. doi: 10.1137/0310041.

[13]

H. J. Kushner, On the stochastic maximum principle: Fixed time of control, J. Math. Anal. Appl., 11 (1965), 78-92. doi: 10.1016/0022-247X(65)90070-3.

[14]

H. J. Kushner and F. C. Schweppe, A maximum principle for stochastic control systems, J. Math. Anal. Appl., 8 (1964), 287-302. doi: 10.1016/0022-247X(64)90070-8.

[15]

L. Mazliak, An algorithm for solving a stochastic control problem, Stochastic analysis and applications, 14 (1996), 513-533. doi: 10.1080/07362999608809455.

[16]

L. Mou and J. Yong, A variational formula for stochastic controls and some applications, Pure Appl. Math. Q., 3 (2007), 539-567. doi: 10.4310/PAMQ.2007.v3.n2.a7.

[17]

S. G. Peng, A general stochastic maximum principle for optimal control problems, SIAM J. Control Optim., 28 (1990), 966-979. doi: 10.1137/0328054.

[18]

L. Pontryagin, V. Boltyanskiĭ, R. Gamkrelidze and E. Mishchenko, The Mathematical Theory of Optimal Processes, Gordon & Breach Science Publishers, New York, 1986, Reprint of the 1962 English translation.

[19]

Y. Sakawa and Y. Shindo, On global convergence of an algorithm for optimal control, IEEE Trans. Automat. Control, 25 (1980), 1149-1153. doi: 10.1109/TAC.1980.1102517.

[20]

J. Yong and X. Zhou, Stochastic Controls, Hamiltonian Systems and HJB Equations, Springer-Verlag, New York, 1999. doi: 10.1007/978-1-4612-1466-3.

[1]

Huaiqiang Yu, Bin Liu. Pontryagin's principle for local solutions of optimal control governed by the 2D Navier-Stokes equations with mixed control-state constraints. Mathematical Control and Related Fields, 2012, 2 (1) : 61-80. doi: 10.3934/mcrf.2012.2.61
[2]

Guy Barles, Ariela Briani, Emmanuel Trélat. Value function for regional control problems via dynamic programming and Pontryagin maximum principle. Mathematical Control and Related Fields, 2018, 8 (3&4) : 509-533. doi: 10.3934/mcrf.2018021
[3]

Stefan Doboszczak, Manil T. Mohan, Sivaguru S. Sritharan. Pontryagin maximum principle for the optimal control of linearized compressible navier-stokes equations with state constraints. Evolution Equations and Control Theory, 2022, 11 (2) : 347-371. doi: 10.3934/eect.2020110
[4]

Feng Bao, Yanzhao Cao, Weidong Zhao. A first order semi-discrete algorithm for backward doubly stochastic differential equations. Discrete and Continuous Dynamical Systems - B, 2015, 20 (5) : 1297-1313. doi: 10.3934/dcdsb.2015.20.1297
[5]

Hancheng Guo, Jie Xiong. A second-order stochastic maximum principle for generalized mean-field singular control problem. Mathematical Control and Related Fields, 2018, 8 (2) : 451-473. doi: 10.3934/mcrf.2018018
[6]

Kai Wang, Deren Han. On the linear convergence of the general first order primal-dual algorithm. Journal of Industrial and Management Optimization, 2021  doi: 10.3934/jimo.2021134
[7]

Xiao-Li Ding, Iván Area, Juan J. Nieto. Controlled singular evolution equations and Pontryagin type maximum principle with applications. Evolution Equations and Control Theory, 2021  doi: 10.3934/eect.2021059
[8]

Shanjian Tang. A second-order maximum principle for singular optimal stochastic controls. Discrete and Continuous Dynamical Systems - B, 2010, 14 (4) : 1581-1599. doi: 10.3934/dcdsb.2010.14.1581
[9]

Shaolin Ji, Xiaole Xue. A stochastic maximum principle for linear quadratic problem with nonconvex control domain. Mathematical Control and Related Fields, 2019, 9 (3) : 495-507. doi: 10.3934/mcrf.2019022
[10]

Christine Burggraf, Wilfried Grecksch, Thomas Glauben. Stochastic control of individual's health investments. Conference Publications, 2015, 2015 (special) : 159-168. doi: 10.3934/proc.2015.0159
[11]

Omid S. Fard, Javad Soolaki, Delfim F. M. Torres. A necessary condition of Pontryagin type for fuzzy fractional optimal control problems. Discrete and Continuous Dynamical Systems - S, 2018, 11 (1) : 59-76. doi: 10.3934/dcdss.2018004
[12]

Zhen Wu, Feng Zhang. Maximum principle for discrete-time stochastic optimal control problem and stochastic game. Mathematical Control and Related Fields, 2022, 12 (2) : 475-493. doi: 10.3934/mcrf.2021031
[13]

Xiaomin Huang, Yanpei Jiang, Wei Liu. Freidlin-Wentzell's large deviation principle for stochastic integral evolution equations. Communications on Pure and Applied Analysis, , () : -. doi: 10.3934/cpaa.2022091
[14]

Hans Josef Pesch. Carathéodory's royal road of the calculus of variations: Missed exits to the maximum principle of optimal control theory. Numerical Algebra, Control and Optimization, 2013, 3 (1) : 161-173. doi: 10.3934/naco.2013.3.161
[15]

Vladimir Kazakov. Sampling - reconstruction procedure with jitter of markov continuous processes formed by stochastic differential equations of the first order. Conference Publications, 2009, 2009 (Special) : 433-441. doi: 10.3934/proc.2009.2009.433
[16]

Ciro D'Apice, Olha P. Kupenko, Rosanna Manzo. On boundary optimal control problem for an arterial system: First-order optimality conditions. Networks and Heterogeneous Media, 2018, 13 (4) : 585-607. doi: 10.3934/nhm.2018027
[17]

Y. Gong, X. Xiang. A class of optimal control problems of systems governed by the first order linear dynamic equations on time scales. Journal of Industrial and Management Optimization, 2009, 5 (1) : 1-10. doi: 10.3934/jimo.2009.5.1
[18]

Yanning Li, Edward Canepa, Christian Claudel. Efficient robust control of first order scalar conservation laws using semi-analytical solutions. Discrete and Continuous Dynamical Systems - S, 2014, 7 (3) : 525-542. doi: 10.3934/dcdss.2014.7.525
[19]

Adolfo Damiano Cafaro, Simone Fiori. Optimization of a control law to synchronize first-order dynamical systems on Riemannian manifolds by a transverse component. Discrete and Continuous Dynamical Systems - B, 2022, 27 (7) : 3947-3969. doi: 10.3934/dcdsb.2021213
[20]

Adel Chala, Dahbia Hafayed. On stochastic maximum principle for risk-sensitive of fully coupled forward-backward stochastic control of mean-field type with application. Evolution Equations and Control Theory, 2020, 9 (3) : 817-843. doi: 10.3934/eect.2020035

2021 Impact Factor: 1.141

Tools

Metrics

  • PDF downloads (166)
  • HTML views (0)
  • Cited by (0)

Other articles
by authors

[Back to Top]

Copyright © 2022 American Institute of Mathematical Sciences | Privacy Policy