AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Tsinghua Science and Technology Article
PDF (2.9 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Open Access

IIN-FFD: Intra-Inter Network for Face Forgery Detection

School of Software, Nanjing University of Information Science and Technology, Nanjing 210044, China
Institute of Artificial Intelligence, Guangzhou University, Guangzhou 510006, China
School of Computer Science, University of Electronic Science and Technology of China, Chengdu 611731, Sichuan, China
School of Computer Science, Hunan University, Changsha 410082, Hunan, China
Show Author Information

Abstract

Since different kinds of face forgeries leave similar forgery traces in videos, learning the common features from different kinds of forged faces would achieve promising generalization ability of forgery detection. Therefore, to accurately detect known forgeries while ensuring high generalization ability of detecting unknown forgeries, we propose an intra-inter network (IIN) for face forgery detection (FFD) in videos with continual learning. The proposed IIN mainly consists of three modules, i.e., intra-module, inter-module, and forged trace masking module (FTMM). Specifically, the intra-module is trained for each kind of face forgeries by supervised learning to extract special features, while the inter-module is trained by self-supervised learning to extract the common features. As a result, the common and special features of the different forgeries are decoupled by the two feature learning modules, and then the decoupled common features can be utlized to achieve high generalization ability for FFD. Moreover, the FTMM is deployed for contrastive learning to further improve detection accuracy. The experimental results on FaceForensic++ dataset demonstrate that the proposed IIN outperforms the state-of-the-arts in FFD. Also, the generalization ability of the IIN verified on DFDC and Celeb-DF datasets demonstrates that the proposed IIN significantly improves the generalization ability for FFD.

Keywords

deep learningneural networksinformation securityimage classficationface forgeryface forgery detection

References

[1]
K. A. Pantserev, The malicious use of AI-based deepfake technology as the new threat to psychological security and political stability, in Cyber Defence in the Age of AI, Smart Societies and Augmented Humanity, H. Jahankhani, S. Kendzierskyj, N. Chelvachandran, and J. Ibarra, eds. Cham, Switzerland: Springer, 2020, pp. 37–55.
Crossref
[2]

X. Ding, Z. Raziei, E. C. Larson, E. V. Olinick, P. Krueger, and M. Hahsler, Swapped face detection using deep learning and subjective assessment, EURASIP J. Inf. Secur., vol. 2020, no. 1, p. 6, 2020.
Crossref Google Scholar
[3]
A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Niebner, Faceforensics: A large-scale video dataset for forgery detection in human faces, arXiv preprint arXiv: 1803.09179, 2018.
[4]
A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Niessner, FaceForensics++: Learning to detect manipulated facial images, in Proc. IEEE/CVF Int. Conf. Computer Vision (ICCV), Seoul, Republic of Korea, 2019, pp. 1–11.
Crossref
[5]
F. Marra, D. Gragnaniello, D. Cozzolino, and L. Verdoliva, Detection of GAN-generated fake images over social networks, in Proc. IEEE Conf. Multimedia Information Processing and Retrieval (MIPR ), Miami, FL, USA, 2018, pp. 384–389.
Crossref
[6]
S. Tariq, S. Lee, H. Kim, Y. Shin, and S. S. Woo, Detecting both machine and human created fake face images in the wild, in Proc. 2nd Int. Workshop on Multimedia Privacy and Security, Toronto, Canada, 2018, pp. 81–87.
Crossref
[7]

J. Wang, S. C. Satapathy, S. Wang, and Y. Zhang, LCCNN: A lightweight customized CNN-based distance education app for COVID-19 recognition, Mob. Netw. Appl., pp. 1–16, 2023.
Crossref Google Scholar
[8]

Y. D. Zhang, V. Govindaraj, and Z. Zhu, FECNet: A neural network and a mobile app for COVID-19 recognition, Mob. Netw. Appl., pp. 1–14, 2023.
Crossref Google Scholar
[9]
X. Wei, S. Liang, N. Chen, and X. Cao, Transferable adversarial attacks for image and video object detection, arXiv preprint arXiv: 1811.12641, 2018.
Crossref
[10]

Z. Guo, G. Yang, J. Chen, and X. Sun, Fake face detection via adaptive manipulation traces extraction network, Comput. Vis. Image Underst., vol. 204, p. 103170, 2021.
Crossref Google Scholar
[11]
X. Xuan, B. Peng, W. Wang, and J. Dong, On the generalization of GAN image forensics, in Proc. Biometric Recognition : 14th Chinese Conf., CCBR 2019, Zhuzhou, China, 2019, pp. 134–141.
Crossref
[12]
M. Du, S. Pentyala, Y. Li, and X. Hu, Towards generalizable forgery detection with locality-aware autoencoder, arXiv preprint arXiv: 1909.05999, 2019.
Crossref
[13]
I. Korshunova, W. Shi, J. Dambre, and L. Theis, Fast face-swap using convolutional neural networks, in Proc. IEEE Int. Conf. Computer Vision (ICCV), Venice, Italy, 2017, pp. 3677–3685.
Crossref
[14]
L. Li, J. Bao, H. Yang, D. Chen, and F. Wen, Advancing high fidelity identity swapping for forgery detection, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 5074–5083.
Crossref
[15]
J. Thies, M. Zollhofer, M. Stamminger, C. Theobalt, and M. Niessner, Face2Face: Real-time face capture and reenactment of RGB videos, in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 2387–2395.
Crossref
[16]
J. Thies, M. Zollhöfer, and M. Nießner, Deferred neural rendering: Image synthesis using neural textures, ACM Trans. Graph., vol. 38, no. 4, p. 66.
Crossref
[17]

O. Fried, A. Tewari, M. Zollhöfer, A. Finkelstein, E. Shechtman, D. B. Goldman, K. Genova, Z. Jin, C. Theobalt, and M. Agrawala, Text-based editing of talking-head video, ACM Trans. Graph., vol. 38, no. 4, pp. 1–14, 2019.
Crossref Google Scholar
[18]

S. Suwajanakorn, S. M. Seitz, and I. Kemelmacher-Shlizerman, Synthesizing Obama: Learning lip sync from audio, ACM Trans. Graph., vol. 36, no. 4, p. 95, 2017.
Crossref Google Scholar
[19]
X. Yang, Y. Li, and S. Lyu, Exposing deep fakes using inconsistent head poses, in Proc. ICASSP 2019 - 2019 IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), Brighton, United Kingdom, 2019, pp. 8261– 8265.
Crossref
[20]
Y. Li, M. C. Chang and S. Lyu, In Ictu Oculi: Exposing AI created fake videos by detecting eye blinking, in Proc. 2018 IEEE Int. Workshop on Information Forensics and Security (WIFS), Hong Kong, China, 2018, pp. 11–13.
Crossref
[21]
Y. Li and S. Lyu, Exposing deep fake videos by detecting face warping artifacts, arXiv preprint arXiv: 1811.00656, 2018.
[22]
Y. Luo, Y. Zhang, J. Yan, and W. Liu, Generalizing face forgery detection with high-frequency features, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021, pp. 16317–16326.
Crossref
[23]
Y. Qian, G. Yin, L. Sheng, Z. Chen, and J. Shao, Thinking in Frequency: Face Forgery Detection by Mining Frequency-Aware Clues, in Lecture Notes in Computer Science, A. Vedaldi, H. Bischof, T. Brox, and J. M. Frahm, eds. 2020, vol. 12357, pp. 86–103.
Crossref
[24]

Q. Gu, S. Chen, T. Yao, Y. Chen, S. Ding, and R. Yi, Exploiting fine-grained face forgery clues via progressive enhancement learning, Proc. AAAI Conf. Artif. Intell., vol. 36, no. 1, pp. 735–743, 2022.
Crossref Google Scholar
[25]
L. Li, J. Bao, T. Zhang, H. Yang, D. Chen, F. Wen, and B. Guo, Face X-ray for more general face forgery detection, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 5001–5010.
Crossref
[26]
H. Liu, X. Li, W. Zhou, Y. Chen, Y. He, H. Xue, W. Zhang, and N. Yu, Spatial-phase shallow learning: Rethinking face forgery detection in frequency domain, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021, pp. 772–781.
Crossref
[27]

C. Miao, Z. Tan, Q. Chu, H. Liu, H. Hu, and N. Yu, F2Trans: High-frequency fine-grained transformer for face forgery detection, IEEE Trans. Inform. Forensic Secur., vol. 18, pp. 1039–1051, 2023.
Crossref Google Scholar
[28]

J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al., Overcoming catastrophic forgetting in neural networks, Proc. Natl. Acad. Sci. U. S. A., vol. 114, no. 13, pp. 3521–3526, 2017.
Crossref Google Scholar
[29]
F. Zenke, B. Poole, and S. Ganguli, Continual learning through synaptic intelligence, in Proc. 34th Int. Conf. Machine Learning, Sydney, Australia, 2017, pp. 3987–3995.
[30]
M. Riemer, I. Cases, R. Ajemian, M. Liu, I. Rish, Y. Tu, and G. Tesauro, Learning to learn without forgetting by maximizing transfer and minimizing interference, arXiv preprint arXiv: 1810.11910, 2018.
[31]
Q. Pham, C. Liu, D. Sahoo, and H. Steven, Contextual transformation networks for online continual learning, in Int. Conf. on Learning Representations, Virtual Event, 2021.
[32]
S. A. Khan and H. Dai, Video transformer for deepfake detection with incremental learning, in Proc. 29th ACM Int. Conf. Multimedia, Virtual Event, China, 2021, pp. 1821–1828.
Crossref
[33]
M. Kim, S. Tariq, and S. S. Woo, CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation, in Proc. 29th ACM Int. Conf. Multimedia, Virtual Event, China, 2021, pp. 337–346.
Crossref
[34]
X. Chen and K. He, Exploring simple Siamese representation learning, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021, pp. 15750–15758.
Crossref
[35]
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, Momentum contrast for unsupervised visual representation learning, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 9729–9738.
Crossref
[36]
J. B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, et al., Bootstrap your own latent a new approach to self-supervised learning, in Proc. 34th Int. Conf. Neural Information Processing Systems, Vancouver, Canada, 2020, pp. 21271–21284.
[37]
D. Erhan, A. Courville, Y. Bengio, and P. Vincent, Why does unsupervised pre-training help deep learning, in Proc. thirteenth Int. Conf. on Artificial Intelligence and Statistics, Sardinia, Italy, 2010, pp. 201–208.
[38]
O. Ronneberger, P. Fischer, and T. Brox, U-net: Convolutional networks for biomedical image segmentation, in Int. Conf. on Medical Image Computing and Computer-Assisted intervention, Cham, Switzerland, 2015, pp. 234–241.
Crossref
[39]

Y. Nirkin, L. Wolf, Y. Keller, and T. Hassner, DeepFake detection based on discrepancies between faces and their context, IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 10, pp. 6111–6121, 2022.
Crossref Google Scholar
[40]

J. Yang, A. Li, S. Xiao, W. Lu, and X. Gao, MTD-net: Learning to detect deepfakes images by multi-scale texture difference, IEEE Trans. Inf. Forensics Secur., vol. 16, pp. 4234–4245, 2021.
Crossref Google Scholar
[41]
R. Cipolla, Y. Gal, and A. Kendall, Multi-task learning using uncertainty to weigh losses for scene geometry and semantics, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 7482–7491.
Crossref
[42]
B. Dolhansky, J. Bitton, B. Pflaum, J. Lu, R. Howes, M. Wang, and C. C. Ferrer, The deepfake detection challenge (dfdc) dataset, arXiv preprint arXiv: 2006.07397, 2020.
[43]
Y. Li, X. Yang, P. Sun, H. Qi, and S. Lyu, Celeb-DF: A large-scale challenging dataset for DeepFake forensics, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 3207–3216.
Crossref
[44]
L. Jiang, R. Li, W. Wu, C. Qian, and C. C. Loy, DeeperForensics-1.0: A large-scale dataset for real-world face forgery detection, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 2889–2898.
Crossref
[45]
L. Li, J. Bao, H. Yang, D. Chen, and F. Wen, Faceshifter: Towards high fidelity and occlusion aware face swapping, arXiv preprint arXiv: 1912.13457, 2019.
[46]
D. Afchar, V. Nozick, J. Yamagishi, and I. Echizen, MesoNet: A compact facial video forgery detection network, in Proc. IEEE Int. Workshop on Information Forensics and Security (WIFS), Hong Kong, China, 2018, pp. 1–7.
Crossref
[47]
R. Wang, J. F. Xu, L. Ma, X. Xie, Y. Huang, J. Wang, and Y. Liu, Fakespotter: A simple yet robust baseline for spotting ai-synthesized fake faces, arXiv preprint arXiv: 1909.06122, 2019.
Crossref
[48]
H. Dang, F. Liu, J. Stehouwer, X. Liu, and A. K. Jain, On the detection of digital face manipulation, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 5781– 5790.
Crossref
[49]
Z. Sun, Y. Han, Z. Hua, N. Ruan, and W. Jia, Improving the efficiency and robustness of deepfakes detection through precise geometric features, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021, pp. 3609–3618.
Crossref
[50]
S. Schwarcz and R. Chellappa, Finding facial forgery artifacts with parts-based detectors, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), Nashville, TN, USA, 2021, pp. 933– 942.
Crossref
[51]

P. Yu, J. Fei, Z. Xia, Z. Zhou, and J. Weng, Improving generalization by commonality learning in face forgery detection, IEEE Trans. Inform. Forensic Secur., vol. 17, pp. 547–558, 2022.
Crossref Google Scholar
[52]
D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv: 1412.6980, 2014.
[53]

L. Van der Maaten and G. Hinton, Visualizing data using t-sne, J. Mach. Learn. Res., vol. 9, no. 11, 2008.
Google Scholar
Tsinghua Science and Technology
Volume 29 Issue 6,
December 2024
Pages 1839-1850
DOI: 10.26599/TST.2024.9010022
Cite this article:
Zhou Q, Zhou Z, Bao Z, et al. IIN-FFD: Intra-Inter Network for Face Forgery Detection. Tsinghua Science and Technology, 2024, 29(6): 1839-1850. https://doi.org/10.26599/TST.2024.9010022

231

Views

17

Downloads

0

Crossref

0

Web of Science

0

Scopus

0

CSCD

Altmetrics
Received: 16 November 2023
Revised: 28 December 2023
Accepted: 21 January 2024
Published: 20 June 2024
© The Author(s) 2024.

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).
Return