AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home International Journal of Crowd Science Article
PDF (2 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Open Access

Graph Convolutional Neural Network Based Emotion Recognition with Brain Functional Connectivity Network

Pengzhi Gao1,2, Xiangwei Zheng1,2( ), Tao Wang1,2, Yuang Zhang1,2
School of Information Science and Engineering, Shandong Normal University, Jinan 250300, China
State Key Laboratory of High-End Server & Storage Technology, Jinan 250300, China
Show Author Information

Abstract

Emotion recognition plays an important role in Human Computer Interaction (HCI) and the evaluation of human behavior based on emotional state is an important research topic. The purpose of emotion recognition is to automatically identify human’s emotional states by analyzing physiological or non-physiological signals. The conventional emotion classification methods cannot comprehensively leverage global and local features which are extracted from Electroencephalogram (EEG) signal generated after being stimulated. Therefore, we propose the graph convolutional neural network based emotion recognition with brain functional connectivity network (GERBN). Firstly, raw EEG data of the public DEAP and SEED datasets is preprocessed and adopted in this study. Secondly, emotion-related brain functional connection pattern is constructed using Phase-Locking Value (PLV) adjacency matrix to measure connectivity between the signals of different EEG channels according to phase synchronization. A novel graph structure is constructed where the EEG electrode channels are defined as the vertex, and the edge is strong connection of the binary brain network. Thirdly, the GERBN model that includes six layers is designed to classify and recognize emotional states on the two-dimensional emotional models of valence and arousal. Finally, extensive experiments are conducted on DEAP and SEED datasets. Experimental results demonstrate that the proposed method can improve classification accuracies, in which average accuracies of 80.43% and 88.47% on DEAP are attained on valence and arousal dimensions, respectively. On the SEED dataset, the accuracy reaches 92.37% higher than some of the other methods.

Keywords

emotion recognitionElectroencephalogram (EEG)graph convolutional neural networkbrain functional connectivity

References

[1]

M. Hassan and F. Wendling, Electroencephalography source connectivity: Aiming for high resolution of brain networks in time and space, IEEE Signal Process. Mag., vol. 35, no. 3, pp. 81–96, 2018.
Crossref Google Scholar
[2]
R. Khosrowabadi, M. Heijnen, A. Wahab, and H. C. Quek, The dynamic emotion recognition system based on functional connectivity of brain regions, in Proc. IEEE Intelligent Vehicles Symp., La Jolla, CA, USA, 2010, pp. 377–381.
Crossref
[3]

W. L. Zheng, J. Y. Zhu, and B. L. Lu, Identifying stable patterns over time for emotion recognition from EEG, IEEE Trans. Affect. Comput., vol. 10, no. 3, pp. 417–429, 2019.
Crossref Google Scholar
[4]

L. Santamaria-Granados, M. Munoz-Organero, G. Ramirez-González, E. Abdulhay, and N. Arunkumar, Using deep convolutional neural network for emotion detection on a physiological signals dataset (AMIGOS), IEEE Access, vol. 7, pp. 57–67, 2018.
Crossref Google Scholar
[5]

X. Chen, X. Zheng, K. Sun, W. Liu, and Y. Zhang, Self-supervised vision transformer-based few-shot learning for facial expression recognition, Inf. Sci., vol. 634, pp. 206–226, 2023.
Crossref Google Scholar
[6]
G. Cao, Y. Ma, X. Meng, Y. Gao, and M. Meng, Emotion recognition based on CNN, in Proc. Chinese Control Conf. (CCC), Guangzhou, China, 2019, pp. 8627–8630.
Crossref
[7]

S. Hwang, K. Hong, G. Son, and H. Byun, Learning CNN features from DE features for EEG-based emotion recognition, Pattern Anal. Appl., vol. 23, no. 3, pp. 1323–1335, 2020.
Crossref Google Scholar
[8]

Y. H. Kwon, S. B. Shin, and S. D. Kim, Electroencephalography based fusion two-dimensional (2D)-convolution neural networks (CNN) model for emotion recognition system, Sensors, vol. 18, no. 5, p. 1383, 2018.
Crossref Google Scholar
[9]

M. N. Dar, M. U. Akram, S. G. Khawaja, and A. N. Pujari, CNN and LSTM-based emotion charting using physiological signals, Sensors, vol. 20, no. 16, p. 4551, 2020.
Crossref Google Scholar
[10]

H. Cui, A. Liu, X. Zhang, X. Chen, K. Wang, and X. Chen, EEG-based emotion recognition using an end-to-end regional-asymmetric convolutional neural network, Knowl. Based Syst., vol. 205, p. 106243, 2020.
Crossref Google Scholar
[11]

F. Shen, G. Dai, G. Lin, J. Zhang, W. Kong, and H. Zeng, EEG-based emotion recognition using 4D convolutional recurrent neural network, Cogn. Neurodyn., vol. 14, no. 6, pp. 815–828, 2020.
Crossref Google Scholar
[12]

P. Li, H. Liu, Y. Si, C. Li, F. Li, X. Zhu, X. Huang, Y. Zeng, D. Yao, Y. Zhang, et al., EEG based emotion recognition by combining functional connectivity network and local activations, IEEE Trans. Biomed. Eng., vol. 66, no. 10, pp. 2869–2881, 2019.
Crossref Google Scholar
[13]

S. Sun, X. Li, J. Zhu, Y. Wang, R. La, X. Zhang, L. Wei, and B. Hu, Graph theory analysis of functional connectivity in major depression disorder with high-density resting state EEG data, IEEE Trans. Neural Syst. Rehabil. Eng., vol. 27, no. 3, pp. 429–439, 2019.
Crossref Google Scholar
[14]
V. Gonuguntla, G. Shafiq, Y. Wang, and K. C. Veluvolu, EEG classification of emotions using emotion-specific brain functional network, in Proc. 37th Annual Int. Conf. IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy, 2015, pp. 2896–2899.
Crossref
[15]
J. Wang, W. Wang, Z. Hou, X. Liang, S. Ren, and L. Peng, Brain functional connectivity analysis and crucial channel selection using channel-wise CNN, in Neural Information Processing, L. Cheng, A. C. S. Leung, and S. Ozawa, eds. Cham, Switzerland: Springer, 2018, pp. 40–49.
Crossref
[16]
X. H. Wang, T. Zhang, X. M. Xu, L. Chen, X. F. Xing, and C. L. P. Chen, EEG emotion recognition using dynamical graph convolutional neural networks and broad learning system, in Proc. IEEE Int. Conf. Bioinformatics and Biomedicine (BIBM), Madrid, Spain, 2018, pp. 1240–1244.
Crossref
[17]

T. Song, W. Zheng, P. Song, and Z. Cui, EEG emotion recognition using dynamical graph convolutional neural networks, IEEE Trans. Affect. Comput., vol. 11, no. 3, pp. 532–541, 2020.
Crossref Google Scholar
[18]

Y. Yin, X. Zheng, B. Hu, Y. Zhang, and X. Cui, EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM, Appl. Soft Comput., vol. 100, p. 106954, 2021.
Crossref Google Scholar
[19]

T. Wang, X. Zheng, L. Zhang, Z. Cui, and C. Xu, A graph-based interpretability method for deep neural networks, Neurocomputing, vol. 555, p. 126651, 2023.
Crossref Google Scholar
[20]
D. Valsesia, G. Fracastoro, and E. Magli, Image denoising with graph-convolutional neural networks, in Proc. IEEE Int. Conf. Image Processing (ICIP), Taipei, China, 2019, pp. 2399–2403.
Crossref
[21]

Y. Dasdemir, E. Yildirim, and S. Yildirim, Analysis of functional brain connections for positive-negative emotions using phase locking value, Cogn. Neurodyn., vol. 11, no. 6, pp. 487–500, 2017.
Crossref Google Scholar
[22]

V. Sakkalis, Review of advanced techniques for the estimation of brain connectivity measured with EEG/MEG, Comput. Biol. Med., vol. 41, no. 12, pp. 1110–1117, 2011.
Crossref Google Scholar
[23]

Z. Wang, Y. Tong, and X. Heng, Phase-locking value based graph convolutional neural networks for emotion recognition, IEEE Access, vol. 7, pp. 93711–93722, 2019.
Crossref Google Scholar
[24]
X. Sun, B. Hu, X. Zheng, Y. Yin, and C. Ji, Emotion classification based on brain functional connectivity network, in Proc. IEEE Int. Conf. Bioinformatics and Biomedicine (BIBM), Seoul, Republic of Korea, 2020, pp. 2082–2089.
Crossref
[25]
F. Chung, Spectral Graph Theory. Providence, RI, USA: American Mathematical Society, 1997.
[26]
M. Defferrard, X. Bresson, and P. Vandergheynst, Convolutional neural networks on graphs with fast localized spectral filtering, arXiv preprint 1606.09375, 2016.
[27]
V. Nair and G. E. Hinton, Rectified linear units improve restricted boltzmann machines, in Proc. 27th Int. Conf. Machine Learning (ICML-10), Haifa, Israel, 2010, pp. 807–814.
[28]

Z. Han, Z. Liu, C. M. Vong, Y. S. Liu, S. Bu, J. Han, and C. L. Philip Chen, Deep spatiality: Unsupervised learning of spatially-enhanced global and local 3D features by deep neural network with coupled softmax, IEEE Trans. Image Process., vol. 27, no. 6, pp. 3049–3063, 2018.
Crossref Google Scholar
[29]

S. Koelstra, C. Muhl, M. Soleymani, J. S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, DEAP: A database for emotion analysis; using physiological signals, IEEE Trans. Affective Comput., vol. 3, no. 1, pp. 18–31, 2012.
Crossref Google Scholar
[30]
R. N. Duan, J. Y. Zhu, and B. L. Lu, Differential entropy feature for EEG-based emotion classification, in Proc. 6th Int. IEEE/EMBS Conf. Neural Engineering (NER), San Diego, CA, USA, 2013, pp. 81–84.
Crossref
[31]

Z. Bian, Q. Li, L. Wang, C. Lu, S. Yin, and X. Li, Relative power and coherence of EEG series are related to amnestic mild cognitive impairment in diabetes, Front. Aging Neurosci., vol. 6, p. 11, 2014.
Crossref Google Scholar
[32]

T. Chen, S. Ju, F. Ren, M. Fan, and Y. Gu, EEG emotion recognition model based on the LIBSVM classifier, Measurement, vol. 164, p. 108047, 2020.
Crossref Google Scholar
[33]

R. Nawaz, K. H. Cheah, H. Nisar, and V. V. Yap, Comparison of different feature extraction methods for EEG-based emotion recognition, Biocybern. Biomed. Eng., vol. 40, no. 3, pp. 910–926, 2020.
Crossref Google Scholar
[34]

F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, The graph neural network model, IEEE Trans. Neural Netw., vol. 20, no. 1, pp. 61–80, 2009.
Crossref Google Scholar
International Journal of Crowd Science
Volume 8 Issue 4,
September 2024
Pages 195-204
DOI: 10.26599/IJCS.2024.9100022
Cite this article:
Gao P, Zheng X, Wang T, et al. Graph Convolutional Neural Network Based Emotion Recognition with Brain Functional Connectivity Network. International Journal of Crowd Science, 2024, 8(4): 195-204. https://doi.org/10.26599/IJCS.2024.9100022

147

Views

8

Downloads

0

Crossref

0

Scopus

Altmetrics
Published: 16 September 2024
© The author(s) 2024.

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).
Return