American Institute of Mathematical Sciences

Advanced
September  2019, 9(3): 571-605. doi: 10.3934/mcrf.2019026

Optimal control and zero-sum games for Markov chains of mean-field type

Salah Eddine Choutri 1, , Boualem Djehiche 1, and  Hamidou Tembine 2,,

1. 

Department of Mathematics, KTH Royal Institute of Technology, 100 44, Stockholm, Sweden
2. 

Learning & Game Theory Laboratory, New York University, 19 Washington Square North New York, NY 10011, USA

* Corresponding author: H. Tembine

Received  October 2017 Revised  November 2018 Published  April 2019

We establish existence of Markov chains of mean-field type with unbounded jump intensities by means of a fixed point argument using the total variation distance. We further show existence of nearly-optimal controls and, using a Markov chain backward SDE approach, we suggest conditions for existence of an optimal control and a saddle-point for respectively a control problem and a zero-sum differential game associated with payoff functionals of mean-field type, under dynamics driven by such Markov chains of mean-field type.

Keywords: Mean-field, nonlinear Markov chain, backward SDEs, optimal control, zero-sum game, saddle point, stochastic maximum principle, thinning.
Mathematics Subject Classification: Primary: 60H10, 60H07; Secondary: 49N90.
Citation: Salah Eddine Choutri, Boualem Djehiche, Hamidou Tembine. Optimal control and zero-sum games for Markov chains of mean-field type. Mathematical Control and Related Fields, 2019, 9 (3) : 571-605. doi: 10.3934/mcrf.2019026
References:
[1]

V. E. Beneš, Existence of optimal stochastic control laws, SIAM J. Control, 9 1971,446-472. doi: 10.1137/0309034.

[2]

P. Brèmaud, Point Processes and Queues: Martingale Dynamics, Springer-Verlag, New York-Berlin, 1981.

[3]

S. N. Cohen and R. J. Elliott, Existence, uniqueness and comparisons for BSDEs in general spaces, Annals of Probability, 40 (2012), 2264-2297.  doi: 10.1214/11-AOP679.

[4]

S. N. Cohen and R. J. Elliott, Stochastic Calculus and Applications, Second edition. Probability and its Applications. Springer, Cham, 2015. doi: 10.1007/978-1-4939-2867-5.

[5]

F. Confortola, M. Fuhrman and J. Jacod, Backward stochastic differential equations driven by a marked point process: an elementary approach, with an application to optimal control, Preprint, arXiv: 1407.0876, [math.PR], 2014.

[6]

S. E. Choutri and H. Tembine, A stochastic maximum principle for markov chains of mean-field type, Games, 9 (2018), Paper No. 84, 21 pp, https://doi.org/10.3390/g9040084. doi: 10.3390/g9040084.

[7]

D. Dawson and X. Zheng, Law of large numbers and central limit theorem for unbounded jump mean-field models, Advances in Applied Mathematics, 12 (1991), 293-326. doi: 10.1016/0196-8858(91)90015-B.

[8]

B. Djehiche and S. Hamadène, Optimal control and zero-sum stochastic differential game problems of mean-field type, Appl Math Optim., 2018, https://doi.org/10.1007/s00245-018-9525-6.

[9]

B. Djehiche and I. Kaj, The rate function for some measure-valued jump processes, The Annals of Probability, 23 (1995), 1414-1438.  doi: 10.1214/aop/1176988190.

[10]

B. Djehiche and A. Schied, Large deviations for hierarchical systems of interacting jump processes, Journal of Theoretical Probability, 11 (1998), 1-24.  doi: 10.1023/A:1021690707556.

[11]

I. Ekeland, On the variational principle, J. Math. Anal. Appl., 47 (1974), 324-353.  doi: 10.1016/0022-247X(74)90025-0.

[12]

N. El Karoui and S. Hamadène, BSDEs and risk-sensitive control, zero-sum and nonzero-sum game problems of stochastic functional differential equations, Stochastic Processes and their Application, 107 (2003), 145-169.  doi: 10.1016/S0304-4149(03)00059-0.

[13]

N. El Karoui, S. Peng and M.-C. Quenez, Backward stochastic differential equations in finance, Mathematical Finance, 7 (1997), 1-71.  doi: 10.1111/1467-9965.00022.

[14]

R. J. Elliott and M. Kohlmann, The variational principle and stochastic optimal control, Stochastics, 3 (1980), 229-241.  doi: 10.1080/17442508008833147.

[15]

S. N. Ethier and T. G. Kurtz, Markov Processes: Characterization and Convergence, John Wiley & Sons, Inc., New York, 1986. doi: 10.1002/9780470316658.

[16]

S. Feng, Large deviations for empirical process of mean-field interacting particle system with unbounded jumps, The Annals of Probability, 22 (1994), 2122-2151.  doi: 10.1214/aop/1176988496.

[17]

S. Feng and X. Zheng, Solutions of a class of nonlinear master equations, Stochastic processes and their applications, 43 (1992), 65-84.  doi: 10.1016/0304-4149(92)90076-3.

[18]

S. Hamadène and J. P. Lepeltier, Backward equations, stochastic control and zero-sum stochastic differential games, Stochastics Stochastics Rep., 54 (1995), 221-231. doi: 10.1080/17442509508834006.

[19]

B. Jourdain, S. Méléard and W. Woyczynski, Nonlinear SDEs driven by Lévy processes and related PDEs, ALEA Lat. Am. J. Probab. Math. Stat., 4 (2008), 1-29. 

[20] V. N. Kolokoltsov, Nonlinear Markov Processes and Kinetic Equations, Cambridge University Press, Cambridge, 2010.  doi: 10.1017/CBO9780511760303.
[21]

V. N. Kolokoltsov, Nonlinear Markov games on a finite state space (mean-field and binary interactions), International Journal of Statistics and Probability, 1 (2012), 77.

[22]

C. Léonard, Some epidemic systems are long range interacting particle systems, Stochastic processes in epidemic systems (eds. J.P. Gabriel et al.), Lecture Notes in Biomathematics, volume 86, 1990, Springer.

[23]

C.Léonard, Large deviations for long range interacting particle systems with jumps, Annales de l'IHP Probabilités et Statistiques, 31 (1995), 289-323.

[24]

G. Nicolis and I. Prigogine, Self Organization in Non-Equilibrium Systems, New York-London-Sydney, 1977.

[25]

K. Oelschläger, A martingale approach to the law of large numbers for weakly interacting stochastic processes, The Annals of Probability, (1984), 458-479.

[26]

E. Pardoux and S. Peng, Adapted Solution of a Backward Stochastic Differential Equation, Systems and Control Letters, 14 (1990), 55-61. doi: 10.1016/0167-6911(90)90082-6.

[27] L. C. G. Rogers and D. Williams, Diffusions, Markov Processes and Martingales-Volume 2: Itô Calculus, Cambridge University Press, Cambridge, 2000.  doi: 10.1017/CBO9781107590120.
[28]

F. Schlögl, Chemical reaction models for non-equilibrium phase transitions, Zeitschrift für Physik, 253 (1972), 147-161.

[29]

A. Sokol and N. R. Hansen, Exponential martingales and changes of measure for counting processes, Stochastic Analysis and Applications, 33 (2015), 823-843. doi: 10.1080/07362994.2015.1040890.

[30]

A.-S. Sznitman, Topics in propagation of chaos, Ecole d'Été de Probabilités de Saint-Flour XIX 1989, 1964 (1991), 165-251. doi: 10.1007/BFb0085169.

show all references

References:
[1]

V. E. Beneš, Existence of optimal stochastic control laws, SIAM J. Control, 9 1971,446-472. doi: 10.1137/0309034.

[2]

P. Brèmaud, Point Processes and Queues: Martingale Dynamics, Springer-Verlag, New York-Berlin, 1981.

[3]

S. N. Cohen and R. J. Elliott, Existence, uniqueness and comparisons for BSDEs in general spaces, Annals of Probability, 40 (2012), 2264-2297.  doi: 10.1214/11-AOP679.

[4]

S. N. Cohen and R. J. Elliott, Stochastic Calculus and Applications, Second edition. Probability and its Applications. Springer, Cham, 2015. doi: 10.1007/978-1-4939-2867-5.

[5]

F. Confortola, M. Fuhrman and J. Jacod, Backward stochastic differential equations driven by a marked point process: an elementary approach, with an application to optimal control, Preprint, arXiv: 1407.0876, [math.PR], 2014.

[6]

S. E. Choutri and H. Tembine, A stochastic maximum principle for markov chains of mean-field type, Games, 9 (2018), Paper No. 84, 21 pp, https://doi.org/10.3390/g9040084. doi: 10.3390/g9040084.

[7]

D. Dawson and X. Zheng, Law of large numbers and central limit theorem for unbounded jump mean-field models, Advances in Applied Mathematics, 12 (1991), 293-326. doi: 10.1016/0196-8858(91)90015-B.

[8]

B. Djehiche and S. Hamadène, Optimal control and zero-sum stochastic differential game problems of mean-field type, Appl Math Optim., 2018, https://doi.org/10.1007/s00245-018-9525-6.

[9]

B. Djehiche and I. Kaj, The rate function for some measure-valued jump processes, The Annals of Probability, 23 (1995), 1414-1438.  doi: 10.1214/aop/1176988190.

[10]

B. Djehiche and A. Schied, Large deviations for hierarchical systems of interacting jump processes, Journal of Theoretical Probability, 11 (1998), 1-24.  doi: 10.1023/A:1021690707556.

[11]

I. Ekeland, On the variational principle, J. Math. Anal. Appl., 47 (1974), 324-353.  doi: 10.1016/0022-247X(74)90025-0.

[12]

N. El Karoui and S. Hamadène, BSDEs and risk-sensitive control, zero-sum and nonzero-sum game problems of stochastic functional differential equations, Stochastic Processes and their Application, 107 (2003), 145-169.  doi: 10.1016/S0304-4149(03)00059-0.

[13]

N. El Karoui, S. Peng and M.-C. Quenez, Backward stochastic differential equations in finance, Mathematical Finance, 7 (1997), 1-71.  doi: 10.1111/1467-9965.00022.

[14]

R. J. Elliott and M. Kohlmann, The variational principle and stochastic optimal control, Stochastics, 3 (1980), 229-241.  doi: 10.1080/17442508008833147.

[15]

S. N. Ethier and T. G. Kurtz, Markov Processes: Characterization and Convergence, John Wiley & Sons, Inc., New York, 1986. doi: 10.1002/9780470316658.

[16]

S. Feng, Large deviations for empirical process of mean-field interacting particle system with unbounded jumps, The Annals of Probability, 22 (1994), 2122-2151.  doi: 10.1214/aop/1176988496.

[17]

S. Feng and X. Zheng, Solutions of a class of nonlinear master equations, Stochastic processes and their applications, 43 (1992), 65-84.  doi: 10.1016/0304-4149(92)90076-3.

[18]

S. Hamadène and J. P. Lepeltier, Backward equations, stochastic control and zero-sum stochastic differential games, Stochastics Stochastics Rep., 54 (1995), 221-231. doi: 10.1080/17442509508834006.

[19]

B. Jourdain, S. Méléard and W. Woyczynski, Nonlinear SDEs driven by Lévy processes and related PDEs, ALEA Lat. Am. J. Probab. Math. Stat., 4 (2008), 1-29. 

[20] V. N. Kolokoltsov, Nonlinear Markov Processes and Kinetic Equations, Cambridge University Press, Cambridge, 2010.  doi: 10.1017/CBO9780511760303.
[21]

V. N. Kolokoltsov, Nonlinear Markov games on a finite state space (mean-field and binary interactions), International Journal of Statistics and Probability, 1 (2012), 77.

[22]

C. Léonard, Some epidemic systems are long range interacting particle systems, Stochastic processes in epidemic systems (eds. J.P. Gabriel et al.), Lecture Notes in Biomathematics, volume 86, 1990, Springer.

[23]

C.Léonard, Large deviations for long range interacting particle systems with jumps, Annales de l'IHP Probabilités et Statistiques, 31 (1995), 289-323.

[24]

G. Nicolis and I. Prigogine, Self Organization in Non-Equilibrium Systems, New York-London-Sydney, 1977.

[25]

K. Oelschläger, A martingale approach to the law of large numbers for weakly interacting stochastic processes, The Annals of Probability, (1984), 458-479.

[26]

E. Pardoux and S. Peng, Adapted Solution of a Backward Stochastic Differential Equation, Systems and Control Letters, 14 (1990), 55-61. doi: 10.1016/0167-6911(90)90082-6.

[27] L. C. G. Rogers and D. Williams, Diffusions, Markov Processes and Martingales-Volume 2: Itô Calculus, Cambridge University Press, Cambridge, 2000.  doi: 10.1017/CBO9781107590120.
[28]

F. Schlögl, Chemical reaction models for non-equilibrium phase transitions, Zeitschrift für Physik, 253 (1972), 147-161.

[29]

A. Sokol and N. R. Hansen, Exponential martingales and changes of measure for counting processes, Stochastic Analysis and Applications, 33 (2015), 823-843. doi: 10.1080/07362994.2015.1040890.

[30]

A.-S. Sznitman, Topics in propagation of chaos, Ecole d'Été de Probabilités de Saint-Flour XIX 1989, 1964 (1991), 165-251. doi: 10.1007/BFb0085169.

[1]

Adel Chala, Dahbia Hafayed. On stochastic maximum principle for risk-sensitive of fully coupled forward-backward stochastic control of mean-field type with application. Evolution Equations and Control Theory, 2020, 9 (3) : 817-843. doi: 10.3934/eect.2020035
[2]

Valery Y. Glizer, Oleg Kelis. Singular infinite horizon zero-sum linear-quadratic differential game: Saddle-point equilibrium sequence. Numerical Algebra, Control and Optimization, 2017, 7 (1) : 1-20. doi: 10.3934/naco.2017001
[3]

Hancheng Guo, Jie Xiong. A second-order stochastic maximum principle for generalized mean-field singular control problem. Mathematical Control and Related Fields, 2018, 8 (2) : 451-473. doi: 10.3934/mcrf.2018018
[4]

René Carmona, Kenza Hamidouche, Mathieu Laurière, Zongjun Tan. Linear-quadratic zero-sum mean-field type games: Optimality conditions and policy optimization. Journal of Dynamics and Games, 2021, 8 (4) : 403-443. doi: 10.3934/jdg.2021023
[5]

Tian Chen, Zhen Wu. A general maximum principle for partially observed mean-field stochastic system with random jumps in progressive structure. Mathematical Control and Related Fields, 2022  doi: 10.3934/mcrf.2022012
[6]

Kehan Si, Zhenda Xu, Ka Fai Cedric Yiu, Xun Li. Open-loop solvability for mean-field stochastic linear quadratic optimal control problems of Markov regime-switching system. Journal of Industrial and Management Optimization, 2021  doi: 10.3934/jimo.2021074
[7]

Zhen Wu, Feng Zhang. Maximum principle for discrete-time stochastic optimal control problem and stochastic game. Mathematical Control and Related Fields, 2022, 12 (2) : 475-493. doi: 10.3934/mcrf.2021031
[8]

Fernando Luque-Vásquez, J. Adolfo Minjárez-Sosa. Average optimal strategies for zero-sum Markov games with poorly known payoff function on one side. Journal of Dynamics and Games, 2014, 1 (1) : 105-119. doi: 10.3934/jdg.2014.1.105
[9]

Xiangxiang Huang, Xianping Guo, Jianping Peng. A probability criterion for zero-sum stochastic games. Journal of Dynamics and Games, 2017, 4 (4) : 369-383. doi: 10.3934/jdg.2017020
[10]

Yufeng Shi, Tianxiao Wang, Jiongmin Yong. Mean-field backward stochastic Volterra integral equations. Discrete and Continuous Dynamical Systems - B, 2013, 18 (7) : 1929-1967. doi: 10.3934/dcdsb.2013.18.1929
[11]

Yinggu Chen, Said HamadÈne, Tingshu Mu. Mean-field doubly reflected backward stochastic differential equations. Numerical Algebra, Control and Optimization, 2022  doi: 10.3934/naco.2022012
[12]

Kai Du, Jianhui Huang, Zhen Wu. Linear quadratic mean-field-game of backward stochastic differential systems. Mathematical Control and Related Fields, 2018, 8 (3&4) : 653-678. doi: 10.3934/mcrf.2018028
[13]

Dejian Chang, Zhen Wu. Stochastic maximum principle for non-zero sum differential games of FBSDEs with impulse controls and its application to finance. Journal of Industrial and Management Optimization, 2015, 11 (1) : 27-40. doi: 10.3934/jimo.2015.11.27
[14]

Jianhui Huang, Xun Li, Jiongmin Yong. A linear-quadratic optimal control problem for mean-field stochastic differential equations in infinite horizon. Mathematical Control and Related Fields, 2015, 5 (1) : 97-139. doi: 10.3934/mcrf.2015.5.97
[15]

Jingrui Sun, Hanxiao Wang. Mean-field stochastic linear-quadratic optimal control problems: Weak closed-loop solvability. Mathematical Control and Related Fields, 2021, 11 (1) : 47-71. doi: 10.3934/mcrf.2020026
[16]

Xun Li, Jingrui Sun, Jiongmin Yong. Mean-field stochastic linear quadratic optimal control problems: closed-loop solvability. Probability, Uncertainty and Quantitative Risk, 2016, 1 (0) : 2-. doi: 10.1186/s41546-016-0002-3
[17]

Sylvain Sorin, Guillaume Vigeral. Reversibility and oscillations in zero-sum discounted stochastic games. Journal of Dynamics and Games, 2015, 2 (1) : 103-115. doi: 10.3934/jdg.2015.2.103
[18]

Antoine Hochart. An accretive operator approach to ergodic zero-sum stochastic games. Journal of Dynamics and Games, 2019, 6 (1) : 27-51. doi: 10.3934/jdg.2019003
[19]

Qingmeng Wei, Zhiyong Yu. Time-inconsistent recursive zero-sum stochastic differential games. Mathematical Control and Related Fields, 2018, 8 (3&4) : 1051-1079. doi: 10.3934/mcrf.2018045
[20]

Beatris A. Escobedo-Trujillo. Discount-sensitive equilibria in zero-sum stochastic differential games. Journal of Dynamics and Games, 2016, 3 (1) : 25-50. doi: 10.3934/jdg.2016002

2020 Impact Factor: 1.284

Tools

Metrics

  • PDF downloads (268)
  • HTML views (607)
  • Cited by (3)

Other articles
by authors

[Back to Top]

Copyright © 2022 American Institute of Mathematical Sciences | Privacy Policy