American Institute of Mathematical Sciences

Advanced
  • Previous Article
    Determining the waveguide conductivity in a hyperbolic equation from a single measurement on the lateral boundary
  • MCRF Home
  • This Issue
  • Next Article
    A sparse Markov chain approximation of LQ-type stochastic control problems
September  2016, 6(3): 391-406. doi: 10.3934/mcrf.2016008

On the convergence of the Sakawa-Shindo algorithm in stochastic control

J. Frédéric Bonnans 1, , Justina Gianatti 2, and  Francisco J. Silva 3,

1. 

INRIA-Saclay and Centre de Mathématiques Appliquées, Ecole Polytechnique and Laboratoire de Finance des Marchés d'Énergie, 91128 Palaiseau, France
2. 

CIFASIS - Centro Internacional Franco Argentino, de Ciencias de la Información y de Sistemas, CONICET - UNR - AMU, S2000EZP Rosario, Argentina
3. 

Institut de recherche XLIM-DMI, UMR-CNRS 7252, Faculté des sciences et techniques, Université de Limoges, 87060 Limoges, France

Received  May 2015 Revised  August 2015 Published  August 2016

We analyze an algorithm for solving stochastic control problems, based on Pontryagin's maximum principle, due to Sakawa and Shindo in the deterministic case and extended to the stochastic setting by Mazliak. We assume that either the volatility is an affine function of the state, or the dynamics are linear. We obtain a monotone decrease of the cost functions as well as, in the convex case, the fact that the sequence of controls is minimizing, and converges to an optimal solution if it is bounded. In a specific case we interpret the algorithm as the gradient plus projection method and obtain a linear convergence rate to the solution.
Keywords: Pontryagin's principle, first order algorithm., Stochastic control.
Mathematics Subject Classification: Primary: 93E20, 49M05; Secondary: 49K45, 93E25, 60H10, 60H3.
Citation: J. Frédéric Bonnans, Justina Gianatti, Francisco J. Silva. On the convergence of the Sakawa-Shindo algorithm in stochastic control. Mathematical Control & Related Fields, 2016, 6 (3) : 391-406. doi: 10.3934/mcrf.2016008
References:
[1]

J. Backhoff and F. J. Silva, Sensitivity results in stochastic optimal control: A Lagrangian perspective,, ESAIM: COCV, ().  doi: 10.1051/cocv/2015039.  Google Scholar

[2]

A. Bensoussan, Lectures on Stochastic Control, Lectures notes in Maths. Vol. 972, Springer-Verlag, Berlin, 1982.  Google Scholar

[3]

A. Bensoussan, Stochastic maximum principle for distributed parameter systems, J. Franklin Inst., 315 (1983), 387-406, doi: 10.1016/0016-0032(83)90059-5.  Google Scholar

[4]

J.-M. Bismut, Linear quadratic optimal stochastic control with random coefficients, SIAM J. Control Optimization, 14 (1976), 419-444. doi: 10.1137/0314028.  Google Scholar

[5]

J.-M. Bismut, Conjugate convex functions in optimal stochastic control, J. Math. Anal. Appl., 44 (1973), 384-404. doi: 10.1016/0022-247X(73)90066-8.  Google Scholar

[6]

J.-M. Bismut, An introductory approach to duality in optimal stochastic control, SIAM Rev., 20 (1978), 62-78. doi: 10.1137/1020004.  Google Scholar

[7]

J. F. Bonnans, On an algorithm for optimal control using Pontryagin's maximum principle, SIAM J. Control Optim., 24 (1986), 579-588. doi: 10.1137/0324034.  Google Scholar

[8]

J. F. Bonnans and F. J. Silva, First and second order necessary conditions for stochastic optimal control problems, Appl. Math. Optim., 65 (2012), 403-439. doi: 10.1007/s00245-012-9162-4.  Google Scholar

[9]

A. Cadenillas and I. Karatzas, The stochastic maximum principle for linear convex optimal control with random coefficients, SIAM J. Control Optim., 33 (1995), 590-624. doi: 10.1137/S0363012992240722.  Google Scholar

[10]

A. Goldstein, Convex programming in Hilbert space, Bull. Amer. Math. Soc., 70 (1964), 709-710. doi: 10.1090/S0002-9904-1964-11178-2.  Google Scholar

[11]

U. G. Haussmann, Some examples of optimal stochastic controls or: The stochastic maximum principle at work, SIAM Rev., 23 (1981), 292-307. doi: 10.1137/1023062.  Google Scholar

[12]

H. J. Kushner, Necessary conditions for continuous parameter stochastic optimization problems, SIAM J. Control, 10 (1972), 550-565. doi: 10.1137/0310041.  Google Scholar

[13]

H. J. Kushner, On the stochastic maximum principle: Fixed time of control, J. Math. Anal. Appl., 11 (1965), 78-92. doi: 10.1016/0022-247X(65)90070-3.  Google Scholar

[14]

H. J. Kushner and F. C. Schweppe, A maximum principle for stochastic control systems, J. Math. Anal. Appl., 8 (1964), 287-302. doi: 10.1016/0022-247X(64)90070-8.  Google Scholar

[15]

L. Mazliak, An algorithm for solving a stochastic control problem, Stochastic analysis and applications, 14 (1996), 513-533. doi: 10.1080/07362999608809455.  Google Scholar

[16]

L. Mou and J. Yong, A variational formula for stochastic controls and some applications, Pure Appl. Math. Q., 3 (2007), 539-567. doi: 10.4310/PAMQ.2007.v3.n2.a7.  Google Scholar

[17]

S. G. Peng, A general stochastic maximum principle for optimal control problems, SIAM J. Control Optim., 28 (1990), 966-979. doi: 10.1137/0328054.  Google Scholar

[18]

L. Pontryagin, V. Boltyanskiĭ, R. Gamkrelidze and E. Mishchenko, The Mathematical Theory of Optimal Processes, Gordon & Breach Science Publishers, New York, 1986, Reprint of the 1962 English translation.  Google Scholar

[19]

Y. Sakawa and Y. Shindo, On global convergence of an algorithm for optimal control, IEEE Trans. Automat. Control, 25 (1980), 1149-1153. doi: 10.1109/TAC.1980.1102517.  Google Scholar

[20]

J. Yong and X. Zhou, Stochastic Controls, Hamiltonian Systems and HJB Equations, Springer-Verlag, New York, 1999. doi: 10.1007/978-1-4612-1466-3.  Google Scholar

show all references

References:
[1]

J. Backhoff and F. J. Silva, Sensitivity results in stochastic optimal control: A Lagrangian perspective,, ESAIM: COCV, ().  doi: 10.1051/cocv/2015039.  Google Scholar

[2]

A. Bensoussan, Lectures on Stochastic Control, Lectures notes in Maths. Vol. 972, Springer-Verlag, Berlin, 1982.  Google Scholar

[3]

A. Bensoussan, Stochastic maximum principle for distributed parameter systems, J. Franklin Inst., 315 (1983), 387-406, doi: 10.1016/0016-0032(83)90059-5.  Google Scholar

[4]

J.-M. Bismut, Linear quadratic optimal stochastic control with random coefficients, SIAM J. Control Optimization, 14 (1976), 419-444. doi: 10.1137/0314028.  Google Scholar

[5]

J.-M. Bismut, Conjugate convex functions in optimal stochastic control, J. Math. Anal. Appl., 44 (1973), 384-404. doi: 10.1016/0022-247X(73)90066-8.  Google Scholar

[6]

J.-M. Bismut, An introductory approach to duality in optimal stochastic control, SIAM Rev., 20 (1978), 62-78. doi: 10.1137/1020004.  Google Scholar

[7]

J. F. Bonnans, On an algorithm for optimal control using Pontryagin's maximum principle, SIAM J. Control Optim., 24 (1986), 579-588. doi: 10.1137/0324034.  Google Scholar

[8]

J. F. Bonnans and F. J. Silva, First and second order necessary conditions for stochastic optimal control problems, Appl. Math. Optim., 65 (2012), 403-439. doi: 10.1007/s00245-012-9162-4.  Google Scholar

[9]

A. Cadenillas and I. Karatzas, The stochastic maximum principle for linear convex optimal control with random coefficients, SIAM J. Control Optim., 33 (1995), 590-624. doi: 10.1137/S0363012992240722.  Google Scholar

[10]

A. Goldstein, Convex programming in Hilbert space, Bull. Amer. Math. Soc., 70 (1964), 709-710. doi: 10.1090/S0002-9904-1964-11178-2.  Google Scholar

[11]

U. G. Haussmann, Some examples of optimal stochastic controls or: The stochastic maximum principle at work, SIAM Rev., 23 (1981), 292-307. doi: 10.1137/1023062.  Google Scholar

[12]

H. J. Kushner, Necessary conditions for continuous parameter stochastic optimization problems, SIAM J. Control, 10 (1972), 550-565. doi: 10.1137/0310041.  Google Scholar

[13]

H. J. Kushner, On the stochastic maximum principle: Fixed time of control, J. Math. Anal. Appl., 11 (1965), 78-92. doi: 10.1016/0022-247X(65)90070-3.  Google Scholar

[14]

H. J. Kushner and F. C. Schweppe, A maximum principle for stochastic control systems, J. Math. Anal. Appl., 8 (1964), 287-302. doi: 10.1016/0022-247X(64)90070-8.  Google Scholar

[15]

L. Mazliak, An algorithm for solving a stochastic control problem, Stochastic analysis and applications, 14 (1996), 513-533. doi: 10.1080/07362999608809455.  Google Scholar

[16]

L. Mou and J. Yong, A variational formula for stochastic controls and some applications, Pure Appl. Math. Q., 3 (2007), 539-567. doi: 10.4310/PAMQ.2007.v3.n2.a7.  Google Scholar

[17]

S. G. Peng, A general stochastic maximum principle for optimal control problems, SIAM J. Control Optim., 28 (1990), 966-979. doi: 10.1137/0328054.  Google Scholar

[18]

L. Pontryagin, V. Boltyanskiĭ, R. Gamkrelidze and E. Mishchenko, The Mathematical Theory of Optimal Processes, Gordon & Breach Science Publishers, New York, 1986, Reprint of the 1962 English translation.  Google Scholar

[19]

Y. Sakawa and Y. Shindo, On global convergence of an algorithm for optimal control, IEEE Trans. Automat. Control, 25 (1980), 1149-1153. doi: 10.1109/TAC.1980.1102517.  Google Scholar

[20]

J. Yong and X. Zhou, Stochastic Controls, Hamiltonian Systems and HJB Equations, Springer-Verlag, New York, 1999. doi: 10.1007/978-1-4612-1466-3.  Google Scholar

[1]

Huaiqiang Yu, Bin Liu. Pontryagin's principle for local solutions of optimal control governed by the 2D Navier-Stokes equations with mixed control-state constraints. Mathematical Control & Related Fields, 2012, 2 (1) : 61-80. doi: 10.3934/mcrf.2012.2.61
[2]

Guy Barles, Ariela Briani, Emmanuel Trélat. Value function for regional control problems via dynamic programming and Pontryagin maximum principle. Mathematical Control & Related Fields, 2018, 8 (3&4) : 509-533. doi: 10.3934/mcrf.2018021
[3]

Stefan Doboszczak, Manil T. Mohan, Sivaguru S. Sritharan. Pontryagin maximum principle for the optimal control of linearized compressible navier-stokes equations with state constraints. Evolution Equations & Control Theory, 2020  doi: 10.3934/eect.2020110
[4]

Feng Bao, Yanzhao Cao, Weidong Zhao. A first order semi-discrete algorithm for backward doubly stochastic differential equations. Discrete & Continuous Dynamical Systems - B, 2015, 20 (5) : 1297-1313. doi: 10.3934/dcdsb.2015.20.1297
[5]

Hancheng Guo, Jie Xiong. A second-order stochastic maximum principle for generalized mean-field singular control problem. Mathematical Control & Related Fields, 2018, 8 (2) : 451-473. doi: 10.3934/mcrf.2018018
[6]

Kai Wang, Deren Han. On the linear convergence of the general first order primal-dual algorithm. Journal of Industrial & Management Optimization, 2021  doi: 10.3934/jimo.2021134
[7]

Shanjian Tang. A second-order maximum principle for singular optimal stochastic controls. Discrete & Continuous Dynamical Systems - B, 2010, 14 (4) : 1581-1599. doi: 10.3934/dcdsb.2010.14.1581
[8]

Shaolin Ji, Xiaole Xue. A stochastic maximum principle for linear quadratic problem with nonconvex control domain. Mathematical Control & Related Fields, 2019, 9 (3) : 495-507. doi: 10.3934/mcrf.2019022
[9]

Christine Burggraf, Wilfried Grecksch, Thomas Glauben. Stochastic control of individual's health investments. Conference Publications, 2015, 2015 (special) : 159-168. doi: 10.3934/proc.2015.0159
[10]

Omid S. Fard, Javad Soolaki, Delfim F. M. Torres. A necessary condition of Pontryagin type for fuzzy fractional optimal control problems. Discrete & Continuous Dynamical Systems - S, 2018, 11 (1) : 59-76. doi: 10.3934/dcdss.2018004
[11]

Zhen Wu, Feng Zhang. Maximum principle for discrete-time stochastic optimal control problem and stochastic game. Mathematical Control & Related Fields, 2021  doi: 10.3934/mcrf.2021031
[12]

Hans Josef Pesch. Carathéodory's royal road of the calculus of variations: Missed exits to the maximum principle of optimal control theory. Numerical Algebra, Control & Optimization, 2013, 3 (1) : 161-173. doi: 10.3934/naco.2013.3.161
[13]

Vladimir Kazakov. Sampling - reconstruction procedure with jitter of markov continuous processes formed by stochastic differential equations of the first order. Conference Publications, 2009, 2009 (Special) : 433-441. doi: 10.3934/proc.2009.2009.433
[14]

Ciro D'Apice, Olha P. Kupenko, Rosanna Manzo. On boundary optimal control problem for an arterial system: First-order optimality conditions. Networks & Heterogeneous Media, 2018, 13 (4) : 585-607. doi: 10.3934/nhm.2018027
[15]

Y. Gong, X. Xiang. A class of optimal control problems of systems governed by the first order linear dynamic equations on time scales. Journal of Industrial & Management Optimization, 2009, 5 (1) : 1-10. doi: 10.3934/jimo.2009.5.1
[16]

Yanning Li, Edward Canepa, Christian Claudel. Efficient robust control of first order scalar conservation laws using semi-analytical solutions. Discrete & Continuous Dynamical Systems - S, 2014, 7 (3) : 525-542. doi: 10.3934/dcdss.2014.7.525
[17]

Adolfo Damiano Cafaro, Simone Fiori. Optimization of a control law to synchronize first-order dynamical systems on Riemannian manifolds by a transverse component. Discrete & Continuous Dynamical Systems - B, 2021  doi: 10.3934/dcdsb.2021213
[18]

Adel Chala, Dahbia Hafayed. On stochastic maximum principle for risk-sensitive of fully coupled forward-backward stochastic control of mean-field type with application. Evolution Equations & Control Theory, 2020, 9 (3) : 817-843. doi: 10.3934/eect.2020035
[19]

Carlo Orrieri. A stochastic maximum principle with dissipativity conditions. Discrete & Continuous Dynamical Systems, 2015, 35 (11) : 5499-5519. doi: 10.3934/dcds.2015.35.5499
[20]

J. R. Ward. Periodic solutions of first order systems. Discrete & Continuous Dynamical Systems, 2013, 33 (1) : 381-389. doi: 10.3934/dcds.2013.33.381

2020 Impact Factor: 1.284

Tools

Metrics

  • PDF downloads (130)
  • HTML views (0)
  • Cited by (0)

Other articles
by authors

[Back to Top]

Copyright © 2021 American Institute of Mathematical Sciences | Privacy Policy