Online learning for supervised dimension reduction

Ning Zhang; Qiang Wu

doi:10.3934/mfc.2019008

Article Contents

2019, Volume 2, Issue 2: 95-106. Doi: 10.3934/mfc.2019008

This issue Previous Article "Reducing the number of dimensions of the possible solution space" as a method for finding the exact solution of a system with a large number of unknowns Next Article Nonconvex mixed matrix minimization

Online learning for supervised dimension reduction

Ning Zhang^1, and
Qiang Wu^2, ,

1.
Computational Science PhD Program, Middle Tennessee State University, 1301 E Main Street, Murfreesboro, TN 37132, USA
2.
Department of Mathematical Sciences, Middle Tennessee State University, 1301 E Main Street, Murfreesboro, TN 37132, USA

^* Corresponding author: Qiang Wu
^* Corresponding author: Qiang Wu

Received: March 2019

Published: July 2019

Abstract Full Text(HTML) Figure(2) Related Papers Cited by

Abstract

Online learning has attracted great attention due to the increasingdemand for systems that have the ability of learning and evolving. When thedata to be processed is also high dimensional and dimension reduction is necessary for visualization or prediction enhancement, online dimension reductionwill play an essential role. The purpose of this paper is to propose a new onlinelearning approach for supervised dimension reduction. Our algorithm is motivated by adapting the sliced inverse regression (SIR), a pioneer and effectivealgorithm for supervised dimension reduction, and making it implementable inan incremental manner. The new algorithm, called incremental sliced inverseregression (ISIR), is able to update the subspace of significant factors with intrinsic lower dimensionality fast and efficiently when new observations come in.We also refine the algorithm by using an overlapping technique and develop anincremental overlapping sliced inverse regression (IOSIR) algorithm. We verifythe effectiveness and efficiency of both algorithms by simulations and real dataapplications.

Keywords:

Mathematics Subject Classification: Primary: 68T05; Secondary: 62H25, 68W27.

Citation:

Full Text(HTML)

Figure 1. Simulation results for model (9). (a) Trace correlation and (b) cumulative calculation time by SIR, ISIR, and IOSIR

Download: Full-size image PowerPoint slide

Figure 2. Mean square errors (MSE) for two real data applications: (a) for Concrete Compressive Strength data and (b) for Cpusmall data

Download: Full-size image PowerPoint slide

Related Papers

Cited by

References

[1]	A. Antoniadis, S. Lambert-Lacroix and F. Leblanc, Effective dimension reduction methods for tumor classification using gene expression data, Bioinformatics, 19 (2003), 563-570. doi: 10.1093/bioinformatics/btg062.
[2]	Z. Bai and X. He, A chi-square test for dimensionality with non-Gaussian data, Journal of Multivariate Analysis, 88 (2004), 109-117. doi: 10.1016/S0047-259X(03)00056-3.
[3]	M. P. Barrios and S. Velilla, A bootstrap method for assessing the dimension of a general regression problem, Statistics & Probability Letters, 77 (2007), 247-255. doi: 10.1016/j.spl.2006.07.020.
[4]	C. Becker and R. Fried, Sliced inverse regression for high-dimensional time series, in Exploratory Data Analysis in Empirical Research, Springer, (2003), 3–11.
[5]	P. N. Belhumeur, J. P. Hespanha and D. J. Kriegman, Eigenfaces vs. Fisherfaces: recognition using class specific linear projection, European Conference on Computer Vision, (1996), 43–58. doi: 10.1007/BFb0015522.
[6]	E. Bura and R. D. Cook, Extending sliced inverse regression: the weighted chi-squared test, Journal of the American Statistical Association, 96 (2001), 996-1003. doi: 10.1198/016214501753208979.
[7]	S. Chandrasekaran, B. S. Manjunath, Y.-F. Wang, J. Winkeler and H. Zhang, An eigenspace update algorithm for image analysis, Graphical Models and Image Processing, 59 (1997), 321-332.
[8]	D. Chu, L.-Z. Zhao, M. K.-P. Ng and X. Wang, Incremental linear discriminant analysis: A fast algorithm and comparisons, IEEE Transactions on Neural Networks and Learning Systems, 26 (2015), 2716-2735. doi: 10.1109/TNNLS.2015.2391201.
[9]	R. D. Cook, Using dimension-reduction subspaces to identify important inputs in models of physical systems, in Proceedings of the section on Physical and Engineering Sciences, American Statistical Association Alexandria, VA, (1994), 18–25.
[10]	R. D. Cook and S. Weisberg, Sliced inverse regression for dimension reduction: Comment, Journal of the American Statistical Association, 86 (1991), 328-332.
[11]	R. D. Cook and and X. Zhang, Fused estimators of the central subspace in sufficient dimension reduction, Journal of the American Statistical Association, 109 (2014), 815-827. doi: 10.1080/01621459.2013.866563.
[12]	J. J. Dai, L. Lieu and D. Rocke, Dimension reduction for classification with gene expression microarray data, Statistical Applications in Genetics and Molecular Biology, 5 (2006), Art. 6, 21 pp. doi: 10.2202/1544-6115.1147.
[13]	L. Elnitski, R. C. Hardison, J. Li, S. Yang, D. Kolbe, P. Eswara, M. J. O'Connor, S. Schwartz, W. Miller and F. Chiaromonte, Distinguishing regulatory DNA from neutral sites, Genome Research, 13 (2003), 64-72. doi: 10.1101/gr.817703.
[14]	L. Ferré, Determining the dimension in sliced inverse regression and related methods, Journal of the American Statistical Association, 93 (1998), 132-140. doi: 10.2307/2669610.
[15]	K. Fukumizu, F. R. Bach and M. I. Jordan, Kernel dimensionality reduction for supervised learning, in NIPS, (2003), 81–88.
[16]	A. Gannoun, S. Girard, C. Guinot and J. Saracco, Sliced inverse regression in reference curves estimation, Computational Statistics & Data Analysis, 46 (2004), 103-122. doi: 10.1016/S0167-9473(03)00141-5.
[17]	Y. A. Ghassabeh, A. Ghavami and H. A. Moghaddam, A new incremental face recognition system, in IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, (2007), 335–340. doi: 10.1109/IDAACS.2007.4488435.
[18]	Y. A. Ghassabeh and H. A. Moghaddam, A new incremental optimal feature extraction method for on-line applications, in International Conference Image Analysis and Recognition, Springer, (2007), 399–410. doi: 10.1007/978-3-540-74260-9_36.
[19]	B. Gu, V. S. Sheng, Z. Wang, D. Ho, S. Osman and S. Li, Incremental learning for v-support vector regression, Neural Networks, 67 (2015), 140-150.
[20]	P. M. Hall, D. A. Marshall and R. F. Martin, Incremental eigenanalysis for classification, BMVC, 98 (1998), 286-295. doi: 10.5244/C.12.29.
[21]	P. M. Hall, D. A. Marshall and R. F. Martin, Merging and splitting eigenspace models, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 (2000), 1042-1049. doi: 10.1109/34.877525.
[22]	P. He, K.-T. Fang and C.-J. Xu, The classification tree combined with SIR and its applications to classification of mass spectra, Journal of Data Science, 1 (2003), 425-445.
[23]	T.-K. Kim, S.-F. Wong, B. Stenger, J. Kittler and R. Cipolla, Incremental linear discriminant analysis using sufficient spanning set approximations, in IEEE Conference on Computer Vision and Pattern Recognition, (2007), 1–8. doi: 10.1109/CVPR.2007.382985.
[24]	M. H. Law and A. K. Jain, Incremental nonlinear dimensionality reduction by manifold learning, IEEE Transactions on Pattern Analysis and Machine Intelligence, 28 (2006), 377-391. doi: 10.1109/TPAMI.2006.56.
[25]	K.-C. Li, Sliced inverse regression for dimension reduction, Journal of the American Statistical Association, 86 (1991), 316-327. doi: 10.1080/01621459.1991.10475035.
[26]	K.-C. Li, On principal hessian directions for data visualization and dimension reduction: Another application of Stein's lemma, Journal of the American Statistical Association, 87 (1992), 1025-1039. doi: 10.1080/01621459.1992.10476258.
[27]	L. Li and H. Li, Dimension reduction methods for microarrays with application to censored survival data, Bioinformatics, 20 (2004), 3406-3412. doi: 10.1093/bioinformatics/bth415.
[28]	L. Li and X. Yin, Sliced inverse regression with regularizations, Biometrics, 64 (2008), 124-131. doi: 10.1111/j.1541-0420.2007.00836.x.
[29]	L.-P. Liu, Y. Jiang and Z.-H. Zhou, Least square incremental linear discriminant analysis, in 2009 Ninth IEEE International Conference on Data Mining, (2009), 298–306. doi: 10.1109/ICDM.2009.78.
[30]	G.-F. Lu, J. Zou and Y. Wang, Incremental learning of complete linear discriminant analysis for face recognition, Knowledge-Based Systems, 31 (2012), 19-27. doi: 10.1016/j.knosys.2012.01.016.
[31]	G. M. Nkiet, Consistent estimation of the dimensionality in sliced inverse regression, Annals of the Institute of Statistical Mathematics, 60 (2008), 257-271. doi: 10.1007/s10463-006-0106-0.
[32]	A. Öztaş, M. Pala, E. Özbay, E. Kanca, N. Caglar and M. A. Bhatti, Predicting the compressive strength and slump of high strength concrete using neural network, Construction and Building Materials, 20 (2006), 769-775.
[33]	S. Pang, S. Ozawa and N. Kasabov, Incremental linear discriminant analysis for classification of data streams, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 35 (2005), 905-914. doi: 10.1109/TSMCB.2005.847744.
[34]	C.-X. Ren and D.-Q. Dai, Incremental learning of bidirectional principal components for face recognition, Pattern Recognition, 43 (2010), 318-330. doi: 10.1016/j.patcog.2009.05.020.
[35]	P. Rodriguez and B. Wohlberg, A matlab implementation of a fast incremental principal component pursuit algorithm for video background modeling, in 2014 IEEE International Conference on Image Processing (ICIP), (2014), 3414–3416. doi: 10.1109/ICIP.2014.7025692.
[36]	J. R. Schott, Determining the dimensionality in sliced inverse regression, Journal of the American Statistical Association, 89 (1994), 141-148. doi: 10.1080/01621459.1994.10476455.
[37]	C. M. Setodji and R. D. Cook, K-means inverse regression, Technometrics, 46 (2004), 421-429. doi: 10.1198/004017004000000437.
[38]	F. X. Song, D. Zhang, Q. Chen and J. Yang, A novel supervised dimensionality reduction algorithm for online image recognition, in Pacific-Rim Symposium on Image and Video Technology, Springer, (2006), 198–207. doi: 10.1007/11949534_20.
[39]	J.-G. Wang, E. Sung and W.-Y. Yau, Incremental two-dimensional linear discriminant analysis with applications to face recognition, Journal of Network and Computer Applications, 33 (2010), 314-322. doi: 10.1016/j.jnca.2009.12.014.
[40]	X. Wang, Incremental and Regularized Linear Discriminant Analysis, Ph.D thesis, National University of Singapore, 2012.
[41]	J. Weng, Y. Zhang and W.-S. Hwang, Candid covariance-free incremental principal component analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, 25 (2003), 1034-1040.
[42]	H.-M. Wu, Kernel sliced inverse regression with applications to classification, Journal of Computational and Graphical Statistics, 17 (2008), 590-610. doi: 10.1198/106186008X345161.
[43]	Q. Wu, S. Mukherjee and F. Liang, Localized sliced inverse regression, Journal of Computational and Graphical Statistics, 19 (2010), 843-860. doi: 10.1198/jcgs.2010.08080.
[44]	Y. Xia, H. Tong, W. Li and L.-X. Zhu, An adaptive estimation of dimension reduction space, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64 (2002), 363-410. doi: 10.1111/1467-9868.03411.
[45]	J. Yan, Z. Lei, D. Yi and S. Z. Li, Towards incremental and large scale face recognition, in 2011 International Joint Conference on Biometrics (IJCB), IEEE, (2011), 1–6.
[46]	J. Ye, Q. Li, H. Xiong, H. Park, R. Janardan and V. Kumar, IDR/QR: An incremental dimension reduction algorithm via QR decomposition, KDD '04 Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, (2004), 364–373. doi: 10.1145/1014052.1014093.
[47]	I.-C. Yeh, Modeling of strength of high-performance concrete using artificial neural networks, Cement and Concrete Research, 28 (1998), 1797-1808. doi: 10.1016/S0008-8846(98)00165-3.
[48]	I.-C. Yeh, Design of high-performance concrete mixture using neural networks and nonlinear programming, Journal of Computing in Civil Engineering, 13 (1999), 36-42. doi: 10.1061/(ASCE)0887-3801(1999)13:1(36).
[49]	I.-C. Yeh, Modeling slump flow of concrete using second-order regressions and artificial neural networks, Cement and Concrete Composites, 29 (2007), 474-480. doi: 10.1016/j.cemconcomp.2007.02.001.
[50]	N. Zhang, Z. Yu and Q. Wu, Overlapping sliced inverse regression for dimension reduction, preprint, arXiv: 1806.08911.
[51]	T. Zhang, W. Ye and Y. Shan, Application of sliced inverse regression with fuzzy clustering for thermal error modeling of CNC machine tool, The International Journal of Advanced Manufacturing Technology, 85 (2016), 2761-2771. doi: 10.1007/s00170-015-8135-6.
[52]	H. Zhao and P. C. Yuen, Incremental linear discriminant analysis for face recognition, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 38 (2008), 210-221. doi: 10.1109/TSMCB.2007.908870.
[53]	H. Zhao, P. C. Yuen and J. T. Kwok, A novel incremental principal component analysis and its application for face recognition, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 36 (2006), 873-886.