AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Computational Visual Media Article
PDF (6.4 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Research Article | Open Access

Multi-scale joint feature network for micro-expression recognition

School of Software, Shandong University, Jinan 250101, China
Show Author Information

Abstract

Micro-expression recognition is a substantivecross-study of psychology and computer science, and it has a wide range of applications (e.g., psychological and clinical diagnosis, emotional analysis, criminal investigation, etc.). However, the subtle and diverse changes in facial muscles make it difficult for existing methods to extract effective features, which limits the improvement of micro-expression recognition accuracy. Therefore, we propose a multi-scale joint feature networkbased on optical flow images for micro-expression recognition. First, we generate an optical flow image that reflects subtle facial motion information. The optical flow image is then fed into the multi-scale joint network for feature extraction and classification. The proposed joint feature module (JFM) integrates features from different layers, which is beneficial for the capture of micro-expression features with different amplitudes. To improve the recognition ability of the model, we also adopt a strategy for fusing the feature prediction results of the three JFMs with the backbone network. Our experimental results show that our method is superior to state-of-the-art methods on three benchmark datasets (SMIC, CASME II, and SAMM) and a combined dataset (3DB).

Keywords

micro-expression recognitionmulti-scale featureoptical flowdeep learning

References

[1]
Haggard, E. A.; Isaacs, K. S. Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of Research in Psychotherapy. The Century Psychology Series. Boston: Springer, 154–165, 1966.
Crossref
[2]
Ekman, P.; Friesen, W. V. Nonverbal leakage and clues to deception. Psychiatry Vol. 32, No. 1, 88–106, 1969.
Crossref Google Scholar
[3]
Ekman, P. METT: Micro expression training tool. CD-ROM. Oakland, 2003.
[4]
Huang, X. H.; Zhao, G. Y.; Hong, X. P.; Zheng, W. M.; Pietikäinen, M. Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns. Neurocomputing Vol. 175, 564–578, 2016.
Crossref Google Scholar
[5]
Polikovsky, S.; Kameda, Y.; Ohta, Y. Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. In: Proceedings of the 3rd International Conference on Imaging for Crime Detection and Prevention, 1–6, 2009.
Crossref
[6]
Zhao, G. Y.; Pietikainen, M. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 29, No. 6, 915–928, 2007.
Crossref Google Scholar
[7]
Liong, S. T.; See, J.; Wong, K.; Phan, R. C. W. Less is more: Micro-expression recognition from video using apex frame. Signal Processing: Image Communication Vol. 62, 82–92, 2018.
Crossref Google Scholar
[8]
Liu, Y.-J.; Zhang, J.-K.; Yan, W.-J.; Wang, S.-J.; Zhao, G.; Fu, X. A main directional mean optical ow feature for spontaneous microexpression recognition. IEEE Transactions on Affective Computing Vol. 7, No. 4, 299–310, 2015.
Crossref Google Scholar
[9]
Xu, F.; Zhang, J. P.; Wang, J. Z. Microexpression identification and categorization using a facial dynamics map. IEEE Transactions on Affective Computing Vol. 8, No. 2, 254–267, 2017.
Crossref Google Scholar
[10]
Khor, H. Q.; See, J.; Phan, R. C. W.; Lin, W. Y. Enriched long-term recurrent convolutional network for facial micro-expression recognition. In: Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition, 667–674, 2018.
Crossref
[11]
Xia, Z. Q.; Hong, X. P.; Gao, X. Y.; Feng, X. Y.; Zhao, G. Y. Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions. IEEE Transactions on Multimedia Vol. 22, No. 3, 626–640, 2019.
Crossref Google Scholar
[12]
Gan, Y. S.; Liong, S. T.; Yau, W. C.; Huang, Y. C.; Tan, L. K. OFF-ApexNet on micro-expression recognition system. Signal Processing: Image Communication Vol. 74, 129–139, 2019.
Crossref Google Scholar
[13]
Khor, H.-Q.; See, J.; Liong, S.-T.; Phan, R. C.; Lin, W. Dual-stream shallow networks for facial microexpression recognition. In: Proceedings of the IEEE International Conference on Image Processing, 36–40, 2019.
Crossref
[14]
Shreve, M.; Godavarthy, S.; Manohar, V.; Goldgof, D.; Sarkar, S. Towards macro- and micro-expression spotting in video using strain patterns. In: Proceedings of the Workshop on Applications of Computer Vision, 1–6, 2009.
Crossref
[15]
Li, X.; Pfister, T.; Huang, X.; Zhao, G.; Pietikäinen, M. A spontaneous micro-expression database: Inducement, collection and baseline. In: Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 1–6, 2013.
Crossref
[16]
Yan, W. J.; Qi, W.; Liu, Y. J.; Wang, S. J.; Fu, X. L. CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces. In: Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 1–7, 2013.
[17]
Yan, W. J.; Li, X.; Wang, S. J.; Zhao, G.; Liu, Y. J.; Chen, Y. H.; Fu, X. CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS One Vol. 9, No. 1, e86041, 2014.
Crossref Google Scholar
[18]
Davison, A. K.; Lansley, C.; Costen, N.; Tan, K.; Yap, M. H. SAMM: A spontaneous micro-facial movement dataset. IEEE Transactions on Affective Computing Vol. 9, No. 1, 116–129, 2016.
Crossref Google Scholar
[19]
Huang, X.; Zhao, G.; Hong, X.; Pietikäinen, M.; Zheng, W. Texture description with completed local quantized patterns. In: Image Analysis. Lecture Notes in Computer Science, Vol. 7944. Kämäräinen, J. K.; Koskela, M. Eds. Springer Berlin Heidelberg, 1–10, 2013.
Crossref
[20]
Wang, Y.; See, J.; Phan, R. C. W.; Oh, Y. H. LBP with six intersection points: Reducing redundant information in LBP-TOP for micro-expression recognition. In: Computer Vision – ACCV 2014. Lecture Notes in Computer Science, Vol. 9003. Cremers, D.; Reid, I.; Saito, H.; Yang, M. H. Eds. Springer Cham, 525–537, 2015.
Crossref
[21]
Ben, X. Y.; Jia, X. T.; Yan, R.; Zhang, X.; Meng, W. X. Learning effective binary descriptors for micro-expression recognition transferred by macro-information. Pattern Recognition Letters Vol. 107, 50–58, 2018.
Crossref Google Scholar
[22]
Wang, S.-J.; Yan, W.-J.; Li, X.; Zhao, G.; Fu, X. Micro-expression recognition using dynamic textures on tensor independent color space. In: Proceedings of the 22nd International Conference on Pattern Recognition, 4678–4683, 2014.
Crossref
[23]
Huang, X.; Wang, S.-J.; Liu, X.; Zhao, G.; Feng, X.; Pietikäinen, M. Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Transactions on Affective Computing Vol. 10, No. 1, 32–47, 2017.
Crossref Google Scholar
[24]
Sun, D.; Roth, S.; Black, M. J. A quantitative analysis of current practices in optical ow estimation and the principles behind them.International Journal of Computer Vision Vol. 106, No. 2, 115–137, 2014.
Crossref Google Scholar
[25]
Zach, C.; Pock, T.; Bischof, H. A duality based approach for realtime TV-L1 optical flow. In: Pattern Recognition. Lecture Notes in Computer Science, Vol. 4713. Hamprecht, F. A.; Schnörr, C.; Jähne, B. Eds. Springer Berlin Heidelberg, 214–223, 2007.
[26]
Liu, Y. J.; Li, B. J.; Lai, Y. K. Sparse MDMO: Learning a discriminative feature for micro-expression recognition. IEEE Transactions on Affective Computing Vol. 12, No. 1, 254–261, 2021.
Google Scholar
[27]
Peng, S.; Huang, H. B.; Chen, W. J.; Zhang, L.; Fang, W. W. More trainable inception-ResNet for face recognition. Neurocomputing Vol. 411, 9–19, 2020.
Crossref Google Scholar
[28]
Wang, S.; Cheng, Z.; Deng, X.; Chang, L.; Duan, F.; Lu, K. Leveraging 3D blendshape for facial expression recognition using CNN. Science China Information Sciences Vol. 63, No. 2, 120114, 2020.
Crossref Google Scholar
[29]
Kim, D. H.; Baddar, W. J.; Ro, Y. M. Micro-expression recognition with expression-state constrained spatio-temporal feature representations. In: Proceedings of the 24th ACM international Conference on Multimedia, 382–386, 2016.
Crossref
[30]
Peng, M.; Wu, Z.; Zhang, Z.; Chen, T. From macro to micro expression recognition: Deep learning on small datasets using transfer learning. In: Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition, 657–661, 2018.
Crossref
[31]
Wang, C. Y.; Peng, M.; Bi, T.; Chen, T. Micro-attention for micro-expression recognition. Neurocomputing Vol. 410, 354–362, 2020.
Crossref Google Scholar
[32]
Wang, S. J.; Li, B. J.; Liu, Y. J.; Yan, W. J.; Ou, X. Y.; Huang, X. H.; Fu, X. Micro-expression recognition with small sample size by transferring long-term convolutional neural network. Neurocomputing Vol. 312, 251–262, 2018.
Crossref Google Scholar
[33]
Wu, H. Y.; Rubinstein, M.; Shih, E.; Guttag, J.; Durand, F.; Freeman, W. Eulerian video magnification for revealing subtle changes in the world.ACM Transactions on Graphics Vol. 31, No. 4, Article No. 65, 2012.
Crossref Google Scholar
[34]
Quang, N. V.; Chun, J.; Tokuyama, T. CapsuleNet for micro-expression recognition. In: Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, 1–7, 2019.
Crossref
[35]
Peng, M.; Wang, C.; Bi, T.; Shi, Y.; Zhou, X.; Chen, T. A novel apex-time network for cross-dataset micro-expression recognition. In: Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction, 1–6, 2019.
Crossref
[36]
Liu, Y. C.; Du, H. M.; Zheng, L.; Gedeon, T. A neural micro-expression recognizer. In: Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, 1–4, 2019.
Crossref
[37]
Zhou, L.; Mao, Q. R.; Xue, L. Y. Dual-inception network for cross-database micro-expression recognition. In: Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, 1–5, 2019.
Crossref
[38]
Liong, S. T.; Gan, Y. S.; See, J.; Khor, H. Q.; Huang, Y. C. Shallow triple stream three-dimensional CNN (STSTNet) for micro-expression recognition. In: Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, 1–5, 2019.
Crossref
[39]
Li, Y. T.; Huang, X. H.; Zhao, G. Y. Can micro-expression be recognized based on single apex frame? In: Proceedings of the 25th IEEE International Conference on Image Processing, 3094–3098, 2018.
[40]
See, J.; Yap, M. H.; Li, J.; Hong, X.; Wang, S.-J. MEGC 2019 – The second facial micro-expressions grand challenge. In: Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, 1–5, 2019.
Crossref
[41]
Krizhevsky, A.; Sutskever, I.; Hinton, G. E. ImageNet classification with deep convolutional neural networks. Communications of the ACM Vol. 60, No. 6, 84–90, 2017.
Crossref Google Scholar
[42]
Le Ngo, A. C.; Phan, R. C. W.; See, J. Spontaneous subtle expression recognition: Imbalanced databases and solutions. In: Computer Vision – ACCV 2014. Lecture Notes in Computer Science, Vol. 9006. Cremers, D.; Reid, I.; Saito, H.; Yang, M. H. Eds. Springer Cham, 33–48, 2015.
Crossref
Computational Visual Media
Volume 7 Issue 3,
September 2021
Pages 407-417
DOI: 10.1007/s41095-021-0217-9
Cite this article:
Li X, Wei G, Wang J, et al. Multi-scale joint feature network for micro-expression recognition. Computational Visual Media, 2021, 7(3): 407-417. https://doi.org/10.1007/s41095-021-0217-9

889

Views

26

Downloads

12

Crossref

8

Web of Science

15

Scopus

2

CSCD

Altmetrics
Received: 25 January 2021
Accepted: 25 February 2021
Published: 16 April 2021
© The Author(s) 2021

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www. editorialmanager.com/cvmj.
Return