Figure 1.
The average performance of facial expression recognition on East-Asian individuals when the ethnic-related features are removed gradually
-
Previous Article
Online optimization for residential PV-ESS energy system scheduling
- MFC Home
- This Issue
-
Next Article
SEMANTIC-RTAB-MAP (SRM): A semantic SLAM system with CNNs on depth images
February
2019, 2(1): 43-53.
doi: 10.3934/mfc.2019004
Eliminating other-race effect for multi-ethnic facial expression recognition
| 1. | Dalian Key Lab of Digital Technology for National Culture, College of Computer Science and Engineering, Dalian Minzu University, Dalian 116600, Liaoning, China |
| 2. | Department of Computing, Curtin University, Kent Street, Perth, WA 6102, Australia |
It has been noticed that the performance of multi-ethnic facial expression recognition is affected by other-race effect significantly. Though this phenomenon has been noticed by psychologists and computer vision researchers for decades, the mechanism of other-race effect is still unknown and few work has been done to compensate or remove this effect. This work proposes an ICA-based method to eliminate the other-race effect in automatic 3D facial expression recognition. Firstly, the depth features are extracted from 3D local facial patches, and independent component analysis is applied to project the features into a subspace in which the projected features are mutually independent. The ethnic-related features and expression-related features are supposed to be separated in ICA subspace. Hence, ethnic-sensitive features are then determined by an entropy-based feature selection method and discarded to depress their influence on facial expression recognition. The proposed method is evaluated on benchmark BU-3DFE database, and the experimental results reveal that the influence caused by other-race effect can be suppressed effectively with the proposed method.
Keywords:
Facial expression recognition,
other-race effect,
3D facial feature extraction,
independent component analysis,
feature selection.
Mathematics Subject Classification:
Primary: 68T10; Secondary: 68U10.
Citation:
Mingliang Xue, Xiaodong Duan, Wanquan Liu. Eliminating other-race effect for multi-ethnic facial expression recognition. Mathematical Foundations of Computing,
2019, 2
(1)
: 43-53.
doi: 10.3934/mfc.2019004
![]()
References:
| [1] |
B. M. Craig, Z. Jing and O. V. Lipp,
Facial race and sex cues have a comparable influence on emotion recognition in chinese and australian participants, Attention Perception and Psychophysics, 79 (2017), 2212-2223.
doi: 10.3758/s13414-017-1364-z. |
| [2] |
M. N. Dailey, G. W. Cottrell, C. Padgett and R. Adolphs,
Empath: A neural network that categorizes facial expressions, Journal of Cognitive Neuroscience, 14 (2002), 1158-1173.
doi: 10.1162/089892902760807177. |
| [3] |
M. N. Dailey, C. Joyce, M. J. Lyons, M. Kamachi, H. Ishi, J. Gyoba and G. W. Cottrell,
Evidence and a computational explanation of cultural differences in facial expression recognition, Emotion, 10 (2010), 874-893.
doi: 10.1037/a0020019. |
| [4] |
C. Darwin and I. J. Rachman, The Expression Of Emotions In Man And Animals, Julian Friedmann, 1979. |
| [5] |
J. R. D'Errico, Understanding gridfit, Information available at: http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do. |
| [6] |
P. Ekman and W. V. Friesen,
Facial action coding system (facs): A technique for the measurement of facial actions, Rivista Di Psichiatria, 47 (1978), 126-138.
|
| [7] |
P. Ekman, E. R. Sorenson and W. V. Friesen,
Pan-cultural elements in facial displays of emotion, Science, 164 (1969), 86-88.
doi: 10.1126/science.164.3875.86. |
| [8] |
G. A. Feingold,
The influence of environment on identification of persons and things, Journal of the American Institute of Criminal Law and Criminology, 5 (1914), 39-51.
doi: 10.2307/1133283. |
| [9] |
S. Fu, H. He and Z.-G. Hou,
Learning race from face: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 36 (2014), 2483-2509.
doi: 10.1109/TPAMI.2014.2321570. |
| [10] |
A. Hyvarinen,
Fast and robust fixed-point algorithms for independent component analysis, IEEE Transactions on Neural Networks, 10 (1999), 626-634.
doi: 10.1109/72.761722. |
| [11] |
R. E. Jack, C. Blais, C. Scheepers, P. G. Schyns and R. Caldara,
Cultural confusions show that facial expressions are not universal, Current Biology Cb, 19 (2009), 1543-1548.
doi: 10.1016/j.cub.2009.07.051. |
| [12] |
R. E. Jack, O. G. B. Garrod, H. Yu, R. Caldara and P. G. Schyns,
Facial expressions of emotion are not culturally universal, Proceedings of the National Academy of Sciences of the United States of America, 109 (2012), 7241-7244.
doi: 10.1073/pnas.1200155109. |
| [13] |
H. Li, J. Sun and L. Chen, Location-sensitive sparse representation of deep normal patterns for expression-robust 3d face recognition, in Biometrics (IJCB), 2017 IEEE International Joint Conference on, IEEE, 2017, 234-242.
doi: 10.1109/BTAS.2017.8272703. |
| [14] |
H. Li, J. Sun, Z. Xu and L. Chen,
Multimodal 2d+ 3d facial expression recognition with deep fusion convolutional neural network, IEEE Transactions on Multimedia, 19 (2017), 2816-2831.
doi: 10.1109/TMM.2017.2713408. |
| [15] |
R. S. Malpass and J. Kravitz,
Recognition for faces of own and other race, Journal of Personality and Social Psychology, 13 (1969), 330-334.
doi: 10.1037/h0028434. |
| [16] |
V. Natu and A. J. O'Toole,
Neural perspectives on the other-race effect, Visual Cognition, 21 (2013), 1081-1095.
doi: 10.1080/13506285.2013.811455. |
| [17] |
P. J. Phillips, F. Jiang, A. Narvekar, J. Ayyad and A. J. O'Toole, An other-race effect for face recognition algorithms, ACM Transactions on Applied Perception (TAP), (2010), 1-12.
doi: 10.6028/NIST.IR.7666. |
| [18] |
J. A. Russell, Cross-cultural similarities and differences in affective processing and expression, in Emotions and Affect in Human Factors and Human-Computer Interaction, Elsevier, 2017, 123-141.
doi: 10.1016/B978-0-12-801851-4.00004-5. |
| [19] |
J. M. Susskind, D. H. Lee, A. Cusi, R. Feiman, W. Grabski and A. K. Anderson,
Expressing fear enhances sensory acquisition, Nature Neuroscience, 11 (2008), 843-850.
doi: 10.1038/nn.2138. |
| [20] |
M. Wang, W. Deng, J. Hu, J. Peng, X. Tao and Y. Huang, Racial faces in-the-wild: Reducing racial bias by deep unsupervised domain adaptation, arXiv preprint, arXiv: 1812.00194. |
| [21] |
M. Xue, X. Duan, J. Zhou, C. Wang, Y. Wang, Z. Li and W. Liu, A computational other-race-effect analysis for 3d facial expression recognition, Chinese Conference on Biometric Recognition, 2016, 483-493.
doi: 10.1007/978-3-319-46654-5_53. |
| [22] |
M. Xue, A. Mian, W. Liu and L. Li, Fully automatic 3d facial expression recognition using local depth features, in IEEE Winter Conference on Applications of Computer Vision, IEEE, 2014, 1096-1103. |
| [23] |
X. Yan, T. J. Andrews, R. Jenkins and A. W. Young,
Cross-cultural differences and similarities underlying other-race effects for facial identity and expression, Quarterly Journal of Experimental Psychology, 69 (2016), 1247-1254.
doi: 10.1080/17470218.2016.1146312. |
| [24] |
L. Yin, X. Wei, Y. Sun, J. Wang and M. J. Rosato, A 3d facial expression database for facial behavior research, in 7th International Conference on Automatic Face and Gesture Recognition (FGR06), IEEE, 2006, 211-216. |
| [25] |
Q. Zhen, D. Huang, Y. Wang and L. Chen, Muscular movement model based automatic 3d facial expression recognition, in International Conference on Multimedia Modeling, Springer, 2015, 522-533.
doi: 10.1007/978-3-319-14445-0_45. |
| [26] |
Q. Zhen, D. Huang, Y. Wang and L. Chen,
Muscular movement model-based automatic 3d/4d facial expression recognition, IEEE Transactions on Multimedia, 18 (2016), 1438-1450.
doi: 10.1109/TMM.2016.2557063. |
show all references
References:
| [1] |
B. M. Craig, Z. Jing and O. V. Lipp,
Facial race and sex cues have a comparable influence on emotion recognition in chinese and australian participants, Attention Perception and Psychophysics, 79 (2017), 2212-2223.
doi: 10.3758/s13414-017-1364-z. |
| [2] |
M. N. Dailey, G. W. Cottrell, C. Padgett and R. Adolphs,
Empath: A neural network that categorizes facial expressions, Journal of Cognitive Neuroscience, 14 (2002), 1158-1173.
doi: 10.1162/089892902760807177. |
| [3] |
M. N. Dailey, C. Joyce, M. J. Lyons, M. Kamachi, H. Ishi, J. Gyoba and G. W. Cottrell,
Evidence and a computational explanation of cultural differences in facial expression recognition, Emotion, 10 (2010), 874-893.
doi: 10.1037/a0020019. |
| [4] |
C. Darwin and I. J. Rachman, The Expression Of Emotions In Man And Animals, Julian Friedmann, 1979. |
| [5] |
J. R. D'Errico, Understanding gridfit, Information available at: http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do. |
| [6] |
P. Ekman and W. V. Friesen,
Facial action coding system (facs): A technique for the measurement of facial actions, Rivista Di Psichiatria, 47 (1978), 126-138.
|
| [7] |
P. Ekman, E. R. Sorenson and W. V. Friesen,
Pan-cultural elements in facial displays of emotion, Science, 164 (1969), 86-88.
doi: 10.1126/science.164.3875.86. |
| [8] |
G. A. Feingold,
The influence of environment on identification of persons and things, Journal of the American Institute of Criminal Law and Criminology, 5 (1914), 39-51.
doi: 10.2307/1133283. |
| [9] |
S. Fu, H. He and Z.-G. Hou,
Learning race from face: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 36 (2014), 2483-2509.
doi: 10.1109/TPAMI.2014.2321570. |
| [10] |
A. Hyvarinen,
Fast and robust fixed-point algorithms for independent component analysis, IEEE Transactions on Neural Networks, 10 (1999), 626-634.
doi: 10.1109/72.761722. |
| [11] |
R. E. Jack, C. Blais, C. Scheepers, P. G. Schyns and R. Caldara,
Cultural confusions show that facial expressions are not universal, Current Biology Cb, 19 (2009), 1543-1548.
doi: 10.1016/j.cub.2009.07.051. |
| [12] |
R. E. Jack, O. G. B. Garrod, H. Yu, R. Caldara and P. G. Schyns,
Facial expressions of emotion are not culturally universal, Proceedings of the National Academy of Sciences of the United States of America, 109 (2012), 7241-7244.
doi: 10.1073/pnas.1200155109. |
| [13] |
H. Li, J. Sun and L. Chen, Location-sensitive sparse representation of deep normal patterns for expression-robust 3d face recognition, in Biometrics (IJCB), 2017 IEEE International Joint Conference on, IEEE, 2017, 234-242.
doi: 10.1109/BTAS.2017.8272703. |
| [14] |
H. Li, J. Sun, Z. Xu and L. Chen,
Multimodal 2d+ 3d facial expression recognition with deep fusion convolutional neural network, IEEE Transactions on Multimedia, 19 (2017), 2816-2831.
doi: 10.1109/TMM.2017.2713408. |
| [15] |
R. S. Malpass and J. Kravitz,
Recognition for faces of own and other race, Journal of Personality and Social Psychology, 13 (1969), 330-334.
doi: 10.1037/h0028434. |
| [16] |
V. Natu and A. J. O'Toole,
Neural perspectives on the other-race effect, Visual Cognition, 21 (2013), 1081-1095.
doi: 10.1080/13506285.2013.811455. |
| [17] |
P. J. Phillips, F. Jiang, A. Narvekar, J. Ayyad and A. J. O'Toole, An other-race effect for face recognition algorithms, ACM Transactions on Applied Perception (TAP), (2010), 1-12.
doi: 10.6028/NIST.IR.7666. |
| [18] |
J. A. Russell, Cross-cultural similarities and differences in affective processing and expression, in Emotions and Affect in Human Factors and Human-Computer Interaction, Elsevier, 2017, 123-141.
doi: 10.1016/B978-0-12-801851-4.00004-5. |
| [19] |
J. M. Susskind, D. H. Lee, A. Cusi, R. Feiman, W. Grabski and A. K. Anderson,
Expressing fear enhances sensory acquisition, Nature Neuroscience, 11 (2008), 843-850.
doi: 10.1038/nn.2138. |
| [20] |
M. Wang, W. Deng, J. Hu, J. Peng, X. Tao and Y. Huang, Racial faces in-the-wild: Reducing racial bias by deep unsupervised domain adaptation, arXiv preprint, arXiv: 1812.00194. |
| [21] |
M. Xue, X. Duan, J. Zhou, C. Wang, Y. Wang, Z. Li and W. Liu, A computational other-race-effect analysis for 3d facial expression recognition, Chinese Conference on Biometric Recognition, 2016, 483-493.
doi: 10.1007/978-3-319-46654-5_53. |
| [22] |
M. Xue, A. Mian, W. Liu and L. Li, Fully automatic 3d facial expression recognition using local depth features, in IEEE Winter Conference on Applications of Computer Vision, IEEE, 2014, 1096-1103. |
| [23] |
X. Yan, T. J. Andrews, R. Jenkins and A. W. Young,
Cross-cultural differences and similarities underlying other-race effects for facial identity and expression, Quarterly Journal of Experimental Psychology, 69 (2016), 1247-1254.
doi: 10.1080/17470218.2016.1146312. |
| [24] |
L. Yin, X. Wei, Y. Sun, J. Wang and M. J. Rosato, A 3d facial expression database for facial behavior research, in 7th International Conference on Automatic Face and Gesture Recognition (FGR06), IEEE, 2006, 211-216. |
| [25] |
Q. Zhen, D. Huang, Y. Wang and L. Chen, Muscular movement model based automatic 3d facial expression recognition, in International Conference on Multimedia Modeling, Springer, 2015, 522-533.
doi: 10.1007/978-3-319-14445-0_45. |
| [26] |
Q. Zhen, D. Huang, Y. Wang and L. Chen,
Muscular movement model-based automatic 3d/4d facial expression recognition, IEEE Transactions on Multimedia, 18 (2016), 1438-1450.
doi: 10.1109/TMM.2016.2557063. |
Figure 2.
The confusion matrix of multi-ethnic facial expression recognition before(a) and after(b) ethnic-related feature elimination based on East-Asian individuals
Figure 3.
The average performance of facial expression recognition on White individuals when the ethnic-related features are removed gradually
Figure 4.
The confusion matrix of multi-ethnic facial expression recognition before(a) and after(b) ethnic-related feature elimination based on White individuals
Table 1.
The ethnicity distribution of BU-3DFE database
| Ethnicity | Sample Size | Number of 3D Faces |
| White | 51 | 1224 |
| East-Asian | 24 | 576 |
| Black | 9 | 216 |
| Hispanic-Latino | 8 | 192 |
| Indian | 6 | 144 |
| Middle-East Asian | 2 | 48 |
| Ethnicity | Sample Size | Number of 3D Faces |
| White | 51 | 1224 |
| East-Asian | 24 | 576 |
| Black | 9 | 216 |
| Hispanic-Latino | 8 | 192 |
| Indian | 6 | 144 |
| Middle-East Asian | 2 | 48 |
| [1] |
Jian Zhao, Fang Deng, Jian Jia, Chunmeng Wu, Haibo Li, Yuan Shi, Shunli Zhang. A new face feature point matrix based on geometric features and illumination models for facial attraction analysis. Discrete and Continuous Dynamical Systems - S, 2019, 12 (4&5) : 1065-1072. doi: 10.3934/dcdss.2019073 |
| [2] |
Qiang Yin, Gongfa Li, Jianguo Zhu. Research on the method of step feature extraction for EOD robot based on 2D laser radar. Discrete and Continuous Dynamical Systems - S, 2015, 8 (6) : 1415-1421. doi: 10.3934/dcdss.2015.8.1415 |
| [3] |
Jianguo Dai, Wenxue Huang, Yuanyi Pan. A category-based probabilistic approach to feature selection. Big Data & Information Analytics, 2018 doi: 10.3934/bdia.2017020 |
| [4] |
Zhi Guo Feng, K. F. Cedric Yiu, K.L. Mak. Feature extraction of the patterned textile with deformations via optimal control theory. Discrete and Continuous Dynamical Systems - B, 2011, 16 (4) : 1055-1069. doi: 10.3934/dcdsb.2011.16.1055 |
| [5] |
Yunmei Lu, Mingyuan Yan, Meng Han, Qingliang Yang, Yanqing Zhang. Privacy preserving feature selection and Multiclass Classification for horizontally distributed data. Mathematical Foundations of Computing, 2018, 1 (4) : 331-348. doi: 10.3934/mfc.2018016 |
| [6] |
Mohamed A. Tawhid, Kevin B. Dsouza. Hybrid binary dragonfly enhanced particle swarm optimization algorithm for solving feature selection problems. Mathematical Foundations of Computing, 2018, 1 (2) : 181-200. doi: 10.3934/mfc.2018009 |
| [7] |
Mohammed Abdulrazaq Kahya, Suhaib Abduljabbar Altamir, Zakariya Yahya Algamal. Improving whale optimization algorithm for feature selection with a time-varying transfer function. Numerical Algebra, Control and Optimization, 2021, 11 (1) : 87-98. doi: 10.3934/naco.2020017 |
| [8] |
Junying Hu, Xiaofei Qian, Jun Pei, Changchun Tan, Panos M. Pardalos, Xinbao Liu. A novel quality prediction method based on feature selection considering high dimensional product quality data. Journal of Industrial and Management Optimization, 2022, 18 (4) : 2977-3000. doi: 10.3934/jimo.2021099 |
| [9] |
Makram Hamouda, Chang-Yeol Jung, Roger Temam. Asymptotic analysis for the 3D primitive equations in a channel. Discrete and Continuous Dynamical Systems - S, 2013, 6 (2) : 401-422. doi: 10.3934/dcdss.2013.6.401 |
| [10] |
Xiaoning Zhu, Zhongyi Li, Jian Sun. Expression recognition method combining convolutional features and Transformer. Mathematical Foundations of Computing, 2022 doi: 10.3934/mfc.2022018 |
| [11] |
Yong Zhou. Remarks on regularities for the 3D MHD equations. Discrete and Continuous Dynamical Systems, 2005, 12 (5) : 881-886. doi: 10.3934/dcds.2005.12.881 |
| [12] |
Hyeong-Ohk Bae, Bum Ja Jin. Estimates of the wake for the 3D Oseen equations. Discrete and Continuous Dynamical Systems - B, 2008, 10 (1) : 1-18. doi: 10.3934/dcdsb.2008.10.1 |
| [13] |
Indranil SenGupta, Weisheng Jiang, Bo Sun, Maria Christina Mariani. Superradiance problem in a 3D annular domain. Conference Publications, 2011, 2011 (Special) : 1309-1318. doi: 10.3934/proc.2011.2011.1309 |
| [14] |
Giovanny Guerrero, José Antonio Langa, Antonio Suárez. Biodiversity and vulnerability in a 3D mutualistic system. Discrete and Continuous Dynamical Systems, 2014, 34 (10) : 4107-4126. doi: 10.3934/dcds.2014.34.4107 |
| [15] |
Jiao He, Rafael Granero-Belinchón. On the dynamics of 3D electrified falling films. Discrete and Continuous Dynamical Systems, 2021, 41 (9) : 4041-4064. doi: 10.3934/dcds.2021027 |
| [16] |
Marko Filipović, Ivica Kopriva. A comparison of dictionary based approaches to inpainting and denoising with an emphasis to independent component analysis learned dictionaries. Inverse Problems and Imaging, 2011, 5 (4) : 815-841. doi: 10.3934/ipi.2011.5.815 |
| [17] |
Danuta Gaweł, Krzysztof Fujarewicz. On the sensitivity of feature ranked lists for large-scale biological data. Mathematical Biosciences & Engineering, 2013, 10 (3) : 667-690. doi: 10.3934/mbe.2013.10.667 |
| [18] |
Lok Ming Lui, Yalin Wang, Tony F. Chan, Paul M. Thompson. Brain anatomical feature detection by solving partial differential equations on general manifolds. Discrete and Continuous Dynamical Systems - B, 2007, 7 (3) : 605-618. doi: 10.3934/dcdsb.2007.7.605 |
| [19] |
Austin Lawson, Tyler Hoffman, Yu-Min Chung, Kaitlin Keegan, Sarah Day. A density-based approach to feature detection in persistence diagrams for firn data. Foundations of Data Science, 2021 doi: 10.3934/fods.2021012 |
| [20] |
Elisa Giesecke, Axel Kröner. Classification with Runge-Kutta networks and feature space augmentation. Journal of Computational Dynamics, 2021, 8 (4) : 495-520. doi: 10.3934/jcd.2021018 |
Impact Factor:
Tools
Metrics
Other articles
by authors
[Back to Top]