American Institute of Mathematical Sciences

Advanced
  • Previous Article
    A Primal-dual algorithm for unfolding neutron energy spectrum from multiple activation foils
  • JIMO Home
  • This Issue
  • Next Article
    Tighter quadratically constrained convex reformulations for semi-continuous quadratic programming
September  2021, 17(5): 2345-2366. doi: 10.3934/jimo.2020072

Principal component analysis with drop rank covariance matrix

Yitong Guo and  Bingo Wing-Kuen Ling ,

School of Information Engineering, Guangdong University of Technology, Guangzhou, 510006, China

* Corresponding author: Bingo Wing-Kuen Ling

Received  November 2019 Revised  December 2019 Published  September 2021 Early access  March 2020

This paper considers the principal component analysis when the covariance matrix of the input vectors drops rank. This case sometimes happens when the total number of the input vectors is very limited. First, it is found that the eigen decomposition of the covariance matrix is not uniquely defined. This implies that different transform matrices could be obtained for performing the principal component analysis. Hence, the generalized form of the eigen decomposition of the covariance matrix is given. Also, it is found that the matrix with its columns being the eigenvectors of the covariance matrix is not necessary to be unitary. This implies that the transform for performing the principal component analysis may not be energy preserved. To address this issue, the necessary and sufficient condition for the matrix with its columns being the eigenvectors of the covariance matrix to be unitary is derived. Moreover, since the design of the unitary transform matrix for performing the principal component analysis is usually formulated as an optimization problem, the necessary and sufficient condition for the first order derivative of the Lagrange function to be equal to the zero vector is derived. In fact, the unitary matrix with its columns being the eigenvectors of the covariance matrix is only a particular case of the condition. Furthermore, the necessary and sufficient condition for the second order derivative of the Lagrange function to be a positive definite function is derived. It is found that the unitary matrix with its columns being the eigenvectors of the covariance matrix does not satisfy this condition. Computer numerical simulation results are given to valid the results.

Keywords: Drop rank covariance matrix, principal component analysis, first order derivative condition, second order derivative condition, unitary condition.
Mathematics Subject Classification: Primary: 58F15, 58F17; Secondary: 53C35.
Citation: Yitong Guo, Bingo Wing-Kuen Ling. Principal component analysis with drop rank covariance matrix. Journal of Industrial & Management Optimization, 2021, 17 (5) : 2345-2366. doi: 10.3934/jimo.2020072
References:
[1]

J. Dehaene, M. Moonen and J. Vandewalle, An improved stochastic gradient algorithm for principal component analysis and subspace tracking, IEEE Transactions on Signal Processing, 45 (1997), 2582-2586.   Google Scholar

[2]

C. L. Fancourt and J. C. Principe, Competitive principal component analysis for locally stationary time series, IEEE Transactions on Signal Processing, 46 (1998), 3068-3081.  doi: 10.1109/78.726819.  Google Scholar

[3]

J. B. O. S. Filho and P. S. R. Diniz, A fixed-point online kernel principal component extraction algorithm, IEEE Transactions on Signal Processing, 65 (2017), 6244-6259.  doi: 10.1109/TSP.2017.2750119.  Google Scholar

[4]

I. A. Guimarães and A. C. Neto, Estimation in polytomous logistic model: Comparison of methods, Journal of Industrial and Management Optimization, Journal of Industrial and Management Optimization, 5 (2009), 239-252.  doi: 10.3934/jimo.2009.5.239.  Google Scholar

[5]

R. He, B. G. Hu, W. S. Zheng and X. W. Kong, Robust principal component analysis based on maximum correntropy criterion, IEEE Transactions on Image Processing, 20 (2011), 1485-1494.  doi: 10.1109/TIP.2010.2103949.  Google Scholar

[6]

S. M. Huang and J. F. Yang, Improved principal component regression for face recognition under illumination variations, IEEE Signal Processing Letters, 19 (2012), 179-182.  doi: 10.1109/LSP.2012.2185492.  Google Scholar

[7]

J. Kang, X. Lin and G. Yang, Research of multi-scale PCA algorithm for face recognition, International Conference on Information and Communications Technologies, ICT, (2015), 1–5. Google Scholar

[8]

M. S. Kang, J. H. Bae, B. S. Kang and K. T. Kim, ISAR cross-range scaling using iterative processing via principal component analysis and bisection algorithm, IEEE Transactions on Signal Processing, 64 (2016), 3909-3918.  doi: 10.1109/TSP.2016.2552511.  Google Scholar

[9]

S. Karamizadeh, S. M. Abdullah, A. A. Manaf and M. Zamani, An overview of principal component analysis, Journal of Signal and Information Processing, 4 (2013), 173-175.  doi: 10.4236/jsip.2013.43B031.  Google Scholar

[10]

H. C. Liu, S. T. Li and L. Y. Fang, Robust object tracking based on principal component analysis and local sparse representation, IEEE Transactions on Instrumentation and Measurement, 64 (2015), 2863-2875.   Google Scholar

[11]

Z. Li, Q. L. Ye, Y. T. Guo, Z. K. Tian, B. W. K. Ling and R. W. K. Lam, Wearable non-invasive blood glucose estimation via empirical mode decomposition based hierarchical multiresolution analysis and random forest, 23rd IEEE International Conference on Digital Signal Processing, DSP, (2018), 1–5. doi: 10.1109/ICDSP.2018.8631545.  Google Scholar

[12]

A. Moradi, J. Razmi, R. Babazadeh and A. Sabbaghnia, An integrated Principal Component Analysis and multi-objective mathematical programming approach to agile supply chain network design under uncertainty, Journal of Industrial and Management Optimization, 15 (2019), 855-879.  doi: 10.3934/jimo.2018074.  Google Scholar

[13]

E. M. Moreno, Non-invasive estimate of blood glucose and blood pressure from a photoplethysmography by means of machine learning techniques, Artificial Intelligence in Medicine, 53 (2011), 127-138.   Google Scholar

[14]

P. P. Markopulos, G. N. Karystinos and D. A. Pados, Optimal algorithms for L1-subpace signal processing, IEEE Transactions on Signal Processing, 62 (2014), 5046-5058.  doi: 10.1109/TSP.2014.2338077.  Google Scholar

[15]

O. Ozgonenel and T. Yalcin, Principal component analysis (PCA) based neural network for motor protection, 10th IET International Conference on Developments in Power System Protection, DPSP, (2010), 1–5. doi: 10.1049/cp.2010.0252.  Google Scholar

[16]

S. Ouyang and Z. Bao, Fast principal component extraction by a weighted information criterion, IEEE Transactions on Signal Processing, 50 (2002), 1994-2002.  doi: 10.1109/TSP.2002.800395.  Google Scholar

[17]

Y. M. Ren, L. Liao and S. J. Maybank, Hyperspectral image spectral-spatial feature extraction via tensor principal component analysis, IEEE Geoscience and Remote Sensing Letters, 14 (2017), 1431-1435.  doi: 10.1109/LGRS.2017.2686878.  Google Scholar

[18]

C. E. Thomaz and G. A. Giraldi, A new ranking method for principal components analysis and its application to face image analysis, Image and Vision Computing, 28 (2009), 902-913.  doi: 10.1016/j.imavis.2009.11.005.  Google Scholar

[19]

G. Tang and A. Nehorai, Constrained Cramér-Rao bound on robust principal component analysis, IEEE Transactions on Signal Processing, 59 (2011), 5070-5076.  doi: 10.1109/TSP.2011.2161984.  Google Scholar

[20]

X. C Xiu, Y. Yang, W.Q. Liu, L.C. Kong and M.J. Shang, An improved total variation regularized RPCA for moving object detection with dynamic background, Journal of Industrial and Management Optimization, 13 (2017), 1-14.  doi: 10.3934/jimo.2019024.  Google Scholar

[21]

S. Y. Yi, Z. H. Lai, Z. Y. He, Y. Cheung and Y. Liu, Joint sparse principal component analysis, Pattern Recognition, 61 (2016), 524-536.  doi: 10.1016/j.patcog.2016.08.025.  Google Scholar

[22]

A. Zare, A. Ozdemir, M. A. Iwen and S. Aviyente, Extension of PCA to higher order data structures: An introduction to tensors, tensor decompositions, and tensor PCA, Proceedings of the IEEE, 106 (2018), 1341-1358.  doi: 10.1109/JPROC.2018.2848209.  Google Scholar

[23]

H. Zou and L. Xue, A selective overview of sparse principal component analysis, Proceedings of the IEEE, 106 (2018), 1311-1320.  doi: 10.1109/JPROC.2018.2846588.  Google Scholar

show all references

References:
[1]

J. Dehaene, M. Moonen and J. Vandewalle, An improved stochastic gradient algorithm for principal component analysis and subspace tracking, IEEE Transactions on Signal Processing, 45 (1997), 2582-2586.   Google Scholar

[2]

C. L. Fancourt and J. C. Principe, Competitive principal component analysis for locally stationary time series, IEEE Transactions on Signal Processing, 46 (1998), 3068-3081.  doi: 10.1109/78.726819.  Google Scholar

[3]

J. B. O. S. Filho and P. S. R. Diniz, A fixed-point online kernel principal component extraction algorithm, IEEE Transactions on Signal Processing, 65 (2017), 6244-6259.  doi: 10.1109/TSP.2017.2750119.  Google Scholar

[4]

I. A. Guimarães and A. C. Neto, Estimation in polytomous logistic model: Comparison of methods, Journal of Industrial and Management Optimization, Journal of Industrial and Management Optimization, 5 (2009), 239-252.  doi: 10.3934/jimo.2009.5.239.  Google Scholar

[5]

R. He, B. G. Hu, W. S. Zheng and X. W. Kong, Robust principal component analysis based on maximum correntropy criterion, IEEE Transactions on Image Processing, 20 (2011), 1485-1494.  doi: 10.1109/TIP.2010.2103949.  Google Scholar

[6]

S. M. Huang and J. F. Yang, Improved principal component regression for face recognition under illumination variations, IEEE Signal Processing Letters, 19 (2012), 179-182.  doi: 10.1109/LSP.2012.2185492.  Google Scholar

[7]

J. Kang, X. Lin and G. Yang, Research of multi-scale PCA algorithm for face recognition, International Conference on Information and Communications Technologies, ICT, (2015), 1–5. Google Scholar

[8]

M. S. Kang, J. H. Bae, B. S. Kang and K. T. Kim, ISAR cross-range scaling using iterative processing via principal component analysis and bisection algorithm, IEEE Transactions on Signal Processing, 64 (2016), 3909-3918.  doi: 10.1109/TSP.2016.2552511.  Google Scholar

[9]

S. Karamizadeh, S. M. Abdullah, A. A. Manaf and M. Zamani, An overview of principal component analysis, Journal of Signal and Information Processing, 4 (2013), 173-175.  doi: 10.4236/jsip.2013.43B031.  Google Scholar

[10]

H. C. Liu, S. T. Li and L. Y. Fang, Robust object tracking based on principal component analysis and local sparse representation, IEEE Transactions on Instrumentation and Measurement, 64 (2015), 2863-2875.   Google Scholar

[11]

Z. Li, Q. L. Ye, Y. T. Guo, Z. K. Tian, B. W. K. Ling and R. W. K. Lam, Wearable non-invasive blood glucose estimation via empirical mode decomposition based hierarchical multiresolution analysis and random forest, 23rd IEEE International Conference on Digital Signal Processing, DSP, (2018), 1–5. doi: 10.1109/ICDSP.2018.8631545.  Google Scholar

[12]

A. Moradi, J. Razmi, R. Babazadeh and A. Sabbaghnia, An integrated Principal Component Analysis and multi-objective mathematical programming approach to agile supply chain network design under uncertainty, Journal of Industrial and Management Optimization, 15 (2019), 855-879.  doi: 10.3934/jimo.2018074.  Google Scholar

[13]

E. M. Moreno, Non-invasive estimate of blood glucose and blood pressure from a photoplethysmography by means of machine learning techniques, Artificial Intelligence in Medicine, 53 (2011), 127-138.   Google Scholar

[14]

P. P. Markopulos, G. N. Karystinos and D. A. Pados, Optimal algorithms for L1-subpace signal processing, IEEE Transactions on Signal Processing, 62 (2014), 5046-5058.  doi: 10.1109/TSP.2014.2338077.  Google Scholar

[15]

O. Ozgonenel and T. Yalcin, Principal component analysis (PCA) based neural network for motor protection, 10th IET International Conference on Developments in Power System Protection, DPSP, (2010), 1–5. doi: 10.1049/cp.2010.0252.  Google Scholar

[16]

S. Ouyang and Z. Bao, Fast principal component extraction by a weighted information criterion, IEEE Transactions on Signal Processing, 50 (2002), 1994-2002.  doi: 10.1109/TSP.2002.800395.  Google Scholar

[17]

Y. M. Ren, L. Liao and S. J. Maybank, Hyperspectral image spectral-spatial feature extraction via tensor principal component analysis, IEEE Geoscience and Remote Sensing Letters, 14 (2017), 1431-1435.  doi: 10.1109/LGRS.2017.2686878.  Google Scholar

[18]

C. E. Thomaz and G. A. Giraldi, A new ranking method for principal components analysis and its application to face image analysis, Image and Vision Computing, 28 (2009), 902-913.  doi: 10.1016/j.imavis.2009.11.005.  Google Scholar

[19]

G. Tang and A. Nehorai, Constrained Cramér-Rao bound on robust principal component analysis, IEEE Transactions on Signal Processing, 59 (2011), 5070-5076.  doi: 10.1109/TSP.2011.2161984.  Google Scholar

[20]

X. C Xiu, Y. Yang, W.Q. Liu, L.C. Kong and M.J. Shang, An improved total variation regularized RPCA for moving object detection with dynamic background, Journal of Industrial and Management Optimization, 13 (2017), 1-14.  doi: 10.3934/jimo.2019024.  Google Scholar

[21]

S. Y. Yi, Z. H. Lai, Z. Y. He, Y. Cheung and Y. Liu, Joint sparse principal component analysis, Pattern Recognition, 61 (2016), 524-536.  doi: 10.1016/j.patcog.2016.08.025.  Google Scholar

[22]

A. Zare, A. Ozdemir, M. A. Iwen and S. Aviyente, Extension of PCA to higher order data structures: An introduction to tensors, tensor decompositions, and tensor PCA, Proceedings of the IEEE, 106 (2018), 1341-1358.  doi: 10.1109/JPROC.2018.2848209.  Google Scholar

[23]

H. Zou and L. Xue, A selective overview of sparse principal component analysis, Proceedings of the IEEE, 106 (2018), 1311-1320.  doi: 10.1109/JPROC.2018.2846588.  Google Scholar

Figure 1.  Plots of the feature values. (a) Sample I.D.: 1-5. (b) Sample I.D.: 6-10. (c) Sample I.D.: 11-15. (d) Sample I.D.: 16-20
Figure Options

Download full-size image

Download as PowerPoint slide

[1]

Muhammad Bilal Riaz, Syed Tauseef Saeed. Comprehensive analysis of integer-order, Caputo-Fabrizio (CF) and Atangana-Baleanu (ABC) fractional time derivative for MHD Oldroyd-B fluid with slip effect and time dependent boundary condition. Discrete & Continuous Dynamical Systems - S, 2021, 14 (10) : 3719-3746. doi: 10.3934/dcdss.2020430
[2]

RazIye Mert, A. Zafer. A necessary and sufficient condition for oscillation of second order sublinear delay dynamic equations. Conference Publications, 2011, 2011 (Special) : 1061-1067. doi: 10.3934/proc.2011.2011.1061
[3]

Dorina Mitrea, Marius Mitrea, Sylvie Monniaux. The Poisson problem for the exterior derivative operator with Dirichlet boundary condition in nonsmooth domains. Communications on Pure & Applied Analysis, 2008, 7 (6) : 1295-1333. doi: 10.3934/cpaa.2008.7.1295
[4]

Christina A. Hollon, Jeffrey T. Neugebauer. Positive solutions of a fractional boundary value problem with a fractional derivative boundary condition. Conference Publications, 2015, 2015 (special) : 615-620. doi: 10.3934/proc.2015.0615
[5]

Li-Li Wan, Chun-Lei Tang. Existence and multiplicity of homoclinic orbits for second order Hamiltonian systems without (AR) condition. Discrete & Continuous Dynamical Systems - B, 2011, 15 (1) : 255-271. doi: 10.3934/dcdsb.2011.15.255
[6]

Shenglan Xie, Maoan Han, Peng Zhu. A posteriori error estimate of weak Galerkin fem for second order elliptic problem with mixed boundary condition. Discrete & Continuous Dynamical Systems - B, 2021, 26 (10) : 5217-5226. doi: 10.3934/dcdsb.2020340
[7]

Makoto Okumura, Takeshi Fukao, Daisuke Furihata, Shuji Yoshikawa. A second-order accurate structure-preserving scheme for the Cahn-Hilliard equation with a dynamic boundary condition. Communications on Pure & Applied Analysis, 2022, 21 (2) : 355-392. doi: 10.3934/cpaa.2021181
[8]

Minoru Murai, Kunimochi Sakamoto, Shoji Yotsutani. Representation formula for traveling waves to a derivative nonlinear Schrödinger equation with the periodic boundary condition. Conference Publications, 2015, 2015 (special) : 878-900. doi: 10.3934/proc.2015.0878
[9]

Percy A. Deift, Thomas Trogdon, Govind Menon. On the condition number of the critically-scaled Laguerre Unitary Ensemble. Discrete & Continuous Dynamical Systems, 2016, 36 (8) : 4287-4347. doi: 10.3934/dcds.2016.36.4287
[10]

Ningyu Sha, Lei Shi, Ming Yan. Fast algorithms for robust principal component analysis with an upper bound on the rank. Inverse Problems & Imaging, 2021, 15 (1) : 109-128. doi: 10.3934/ipi.2020067
[11]

Hui Zhang, Jian-Feng Cai, Lizhi Cheng, Jubo Zhu. Strongly convex programming for exact matrix completion and robust principal component analysis. Inverse Problems & Imaging, 2012, 6 (2) : 357-372. doi: 10.3934/ipi.2012.6.357
[12]

Oliver Kolb, Simone Göttlich, Paola Goatin. Capacity drop and traffic control for a second order traffic model. Networks & Heterogeneous Media, 2017, 12 (4) : 663-681. doi: 10.3934/nhm.2017027
[13]

Doyoon Kim, Seungjin Ryu. The weak maximum principle for second-order elliptic and parabolic conormal derivative problems. Communications on Pure & Applied Analysis, 2020, 19 (1) : 493-510. doi: 10.3934/cpaa.2020024
[14]

Hamid Maarouf. Local Kalman rank condition for linear time varying systems. Mathematical Control & Related Fields, 2021  doi: 10.3934/mcrf.2021029
[15]

Qingshan You, Qun Wan, Yipeng Liu. A short note on strongly convex programming for exact matrix completion and robust principal component analysis. Inverse Problems & Imaging, 2013, 7 (1) : 305-306. doi: 10.3934/ipi.2013.7.305
[16]

Fangfang Dong, Yunmei Chen. A fractional-order derivative based variational framework for image denoising. Inverse Problems & Imaging, 2016, 10 (1) : 27-50. doi: 10.3934/ipi.2016.10.27
[17]

Pavel I. Naumkin, Isahi Sánchez-Suárez. Asymptotics for the higher-order derivative nonlinear Schrödinger equation. Communications on Pure & Applied Analysis, 2021, 20 (4) : 1447-1478. doi: 10.3934/cpaa.2021028
[18]

Qi An, Chuncheng Wang, Hao Wang. Analysis of a spatial memory model with nonlocal maturation delay and hostile boundary condition. Discrete & Continuous Dynamical Systems, 2020, 40 (10) : 5845-5868. doi: 10.3934/dcds.2020249
[19]

Xiaofei Cao, Guowei Dai. Stability analysis of a model on varying domain with the Robin boundary condition. Discrete & Continuous Dynamical Systems - S, 2017, 10 (5) : 935-942. doi: 10.3934/dcdss.2017048
[20]

Simão P. S. Santos, Natália Martins, Delfim F. M. Torres. Variational problems of Herglotz type with time delay: DuBois--Reymond condition and Noether's first theorem. Discrete & Continuous Dynamical Systems, 2015, 35 (9) : 4593-4610. doi: 10.3934/dcds.2015.35.4593

2020 Impact Factor: 1.801

Tools

Metrics

  • PDF downloads (261)
  • HTML views (674)
  • Cited by (0)

Other articles
by authors

[Back to Top]

Copyright © 2022 American Institute of Mathematical Sciences | Privacy Policy