Subgradient-based  neural network for nonconvex optimization problems in support vector machines with indefinite kernels

Fengqiu Liu; Xiaoping Xue

doi:10.3934/jimo.2016.12.285

Article Contents

2016, Volume 12, Issue 1: 285-301. Doi: 10.3934/jimo.2016.12.285

This issue Previous Article Quadratic optimization over a polyhedral cone Next Article Finite-time stabilization and $H_\infty$ control of nonlinear delay systems via output feedback

Subgradient-based neural network for nonconvex optimization problems in support vector machines with indefinite kernels

Fengqiu Liu^1, and
Xiaoping Xue^2,

1.
Department of Mathematics, Harbin University of Science and Technology, Harbin, 150080
2.
Department of Mathematics, Harbin Institute of Technology, Harbin 150001

Received: June 2012

Revised: January 2015

Published: January 2016

Abstract Related Papers Cited by

Abstract

Support vector machines (SVMs) with positive semidefinite kernels yield convex quadratic programming problems. SVMs with indefinite kernels yield nonconvex quadratic programming problems. Most optimization methods for SVMs rely on the convexity of objective functions and are not efficient for solving such nonconvex problems. In this paper, we propose a subgradient-based neural network (SGNN) for the problems cast by SVMs with indefinite kernels. It is shown that the state of the proposed neural network has finite length, and as a consequence it converges toward a singleton. The coincidence between the solution and the slow solution of SGNN is also proved starting from the initial value of SGNN. Moreover, we employ the Łojasiewicz inequality to exploit the convergence rate of trajectory of SGNN. The obtained results show that each trajectory is either exponentially convergent, or convergent in finite time, toward a singleton belonging to the set of constrained critical points through a quantitative evaluation of the Łojasiewicz exponent at the equilibrium points. This method is easy to implement without adding any new parameters. Three benchmark data sets from the University of California, Irvine machine learning repository are used in the numerical tests. Experimental results show the efficiency of the proposed neural network.

Keywords:

Mathematics Subject Classification: Primary: 58F15, 58F17; Secondary: 53C35.

Citation:

Related Papers

Cited by

References

[1]	J. P. Aubin and A. Cellina, Differential Inclusions, Springer-Verlag, Berlin, 1984.doi: 10.1007/978-3-642-69512-4.
[2]	J. P. Aubin and H. Frankowska, Set-Valued Analysis, MA: Birkauser, Boston, 1990.
[3]	E. K. P. Chong, S. Hui and S. H. Zak, An analysis of a class of neural networks for solving linear programming problems, Automatic Control, IEEE Transactions on, 44 (1999), 1995-2006.doi: 10.1109/9.802909.
[4]	J. Chen and J. Ye, Training SVM with indefinite kernels, In Proceedings of the 25th International Conference on Machine Learning, (2008), 136-143.doi: 10.1145/1390156.1390174.
[5]	F. H. Clarke, Optimization and Non-smooth Analysis, Wiley, New York, 1969.
[6]	L. Fengqiu and X. Xiaoping, Design of natural classification kernels using prior knowledge, Fuzzy Systems, IEEE Transactions on, 20 (2012), 135-152.
[7]	A. F. Filippove, Differential Equations with Discontinuous Right-Hand Side, Mathematics and Its Applications (Soviet Series), Boston, MA: Kluwer, 1988.
[8]	M. Forti, P. Nistri and M. Quincampoix, Convergence of neural networks for programming problems via a nonsmooth Łojasiewicz inequality, Neural Networks, IEEE Transactions on, 17 (2006), 1471-1486.
[9]	M. Forti and A. Tesi, Absolute stability of analytic neural networks: An approach based on finite trajectory length, Circuits System I, IEEE Transation on, 51 (2004), 2460-2469.doi: 10.1109/TCSI.2004.838143.
[10]	B. Haasdonk, Feature space interpretation of svms with indefinite kernels, Pattern Analysis and Machine Intelligence, IEEE Transactions on, 27 (2005), 482-492.doi: 10.1109/TPAMI.2005.78.
[11]	M. Hein and O. Bousquet, Hilbertian metrics and positive definite kernels on probability measures, In Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics, (eds. Z. Ghahramani and R. Cowell), Society for Artificial Intelligence and Statistics, United States, (2005), 136-143.
[12]	J. J. Hopfield and D. W. Tank, "Neural" computation of decisions in optimization problems, Biological cybernetics, 52 (1985), 141-152.
[13]	R. Luss and A. d'Aspremon, Support vector machine classification with indefinite kernels, Mathematical Programming Computation, 1 (2009), 97-118.doi: 10.1007/s12532-009-0005-5.
[14]	I. Mierswa, Making Indefinite Kernel Learning Practical, Technical report, Artificial Intelligence Unit Department of Computer Science University of Dortmund, 2006.
[15]	J. C. Platt, Fast training of support vector machines using sequential minimal optimization, In Advances in Kernel Methods, (eds. Schölkopf, C. Burges and A. Smola), MIT, (1999), 185-208.
[16]	T. Rockafellar and R. Wets, Variational Analysis, Springer-Verlag, Germany, 1998.doi: 10.1007/978-3-642-02431-3.
[17]	S. Shalev-Shwartz, Y. Singer, N. Srebro and A. Cotter, Pegasos: Primal estimated sub-gradient solver for SVM, Math. Program., 127 (2011), 3-30.doi: 10.1007/s10107-010-0420-4.
[18]	H. Saigo, J. P. Vert, N. Ueda and T. Akutsu, Protein homology detection using string alignment kernels, Bioinformatics, 20 (2004), 1682-1689.doi: 10.1093/bioinformatics/bth141.
[19]	B. Schölkopf and A. J. Smola, Learning with Kernels, MIT, Cambridge, MA, 2002.
[20]	B. Schölkopf, P. Simard, A. J. Smola and V. N. Vapnik, Prior Knowledge in Support Vector Kernels, MIT, Cambridge, MA, 1998.
[21]	V. N. Vapnik, Statistical Learning Theory, Wiley, New York, 1998.
[22]	B. Wei and X. Xiaoping, Subgradient-based neural networks for nonsmooth nonconvex optimization problems, Circuits and Systems I: Regular Papers, IEEE Transactions on, 55 (2008), 2378-2391.doi: 10.1109/TCSI.2008.920131.
[23]	Y. Xia and J. Wang, A one-layer recurrent neural network for support vector machine learning, Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on, 34 (2004), 1621-1629.doi: 10.1109/TSMCB.2003.822955.
[24]	S. Ziye and J. Qingwei, Second order optimality conditions and reformulations for nonconvex quadratically constrained quadratic programming problems, Journal of Industrial and Management Optimization, 10 (2014), 871-882.doi: 10.3934/jimo.2014.10.871.