AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Computational Visual Media Article
PDF (21.9 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Research Article | Open Access

Real-time face view correction for front-facing cameras

University of Science and Technology of China, Hefei 230026, China
School of Computer Science and Informatics, CardiffUniversity, Cardiff, Wales, UK
Show Author Information

Abstract

Face views are particularly important in person-to-person communication. Differenes between the camera location and the face orientation can result in undesirable facial appearances of the participants during video conferencing. This phenomenon is par-ticularly noticeable when using devices where the front-facing camera is placed in unconventional locations such as below the display or within the keyboard. In this paper, we take a video stream from a single RGB camera as input, and generate a video stream that emulates the view from a virtual camera at a designated location. The most challenging issue in this problem is that the corrected view often needs out-of-plane head rotations. To address this challenge, we reconstruct the 3D face shape and re-render it into synthesized frames according to the virtual camera location. To output the corrected video stream with natural appearance in real time, we propose several novel techniques including accurate eyebrow reconstruction, high-quality blending between the corrected face image and background, and template-based 3D reconstruction of glasses. Our system works well for different lighting conditions and skin tones, and can handle users wearing glasses. Extensive experiments and user studies demonstrate that our method provides high-quality results.

Keywords

face view correction3D face reconstruction;deep learningonline communication

Electronic Supplementary Material

Video
CVM_2021_4_437-452_ESM.mp4

References

[1]
Monk, A. F.; Gale, C. A look is worth a thousand words: Full gaze awareness in video-mediated conversation. Discourse Processes Vol. 33, No. 3, 257-278, 2002.
Crossref Google Scholar
[2]
Grayson, D. M.; Monk, A. F. Are You looking at me? Eye contact and desktop video conferencing. ACM Transactions on Computer-Human Interaction Vol. 10, No. 3, 221-243, 2003.
Crossref Google Scholar
[3]
Mukawa, N.; Oka, T.; Arai, K.; Yuasa, M. What is connected by mutual gaze: User’s behavior in video-mediated communication. In: Proceedings of the Extended Abstracts on Human Factors in Computing Systems, 1677-1680, 2005.
Crossref
[4]
Ishii, H.; Kobayashi, M. ClearBoard: A seamless medium for shared drawing and conversation with eye contact. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 525-532,1992.
Crossref
[5]
Okada, K. I.; Maeda, F.; Ichikawaa, Y.; Matsushita, Y. Multiparty videoconferencing at virtual social distance: MAJIC design. In: Proceedings of the ACM Conference on Computer Supported Cooperative Work, 385-393, 1994.
Crossref
[6]
Matusik, W.; Buehler, C.; Raskar, R.; Gortler, S. J.; McMillan, L. Image-based visual hulls. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, 369-374, 2000.
Crossref
[7]
Matusik, W.; Pfister, H. 3D TV: A scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Transactions on Graphics Vol. 23, No. 3, 814-824, 2004.
Crossref Google Scholar
[8]
Kuster, C.; Popa, T.; Zach, C.; Gotsman, C.; Gross, M. H. FreeCam: A hybrid camera system for interactive free-viewpoint video. In: Vision, Modeling, and Visualization. Eisert, P.; Polthier, K.; Hornegger J. Eds. The Eurographics Association, 17-24, 2011.
[9]
Kuster, C.; Popa, T.; Bazin, J. C.; Gotsman, C.; Gross, M. Gaze correction for home video conferencing. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 174, 2012.
Crossref Google Scholar
[10]
Giger, D.; Bazin, J. C.; Kuster, C.; Popa, T.; Gross, M. Gaze correction with a single webcam. In: Proceedings of the IEEE International Conference on Multimedia and Expo, 1-6, 2014.
Crossref
[11]
Kononenko, D.; Lempitsky, V. Learning to look up: Realtime monocular gaze correction using machine learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4667-4675, 2015.
Crossref
[12]
Hsu, C. F.; Wang, Y. S.; Lei, C. L.; Chen, K. T. Look at me! Correcting eye gaze in live video communication. ACM Transactions on Multimedia Computing, Communications, and Applications Vol. 15, No. 2, Article No. 38, 2019.
Crossref Google Scholar
[13]
He, Z.; Spurr, A.; Zhang, X. C.; Hilliges, O. Photo-realistic monocular gaze redirection using generative adversarial networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 6931-6940, 2019.
Crossref
[14]
Zhang, J. C.; Chen, J. J.; Tang, H.; Wang, W.; Yan, Y.; Sangineto, E., Sebe, N. Dual in-painting model for unsupervised gaze correction and animation in the wild. In: Proceedings of the 28th ACM International Conference on Multimedia, 1588-1596, 2020.
Crossref
[15]
Thies, J.; Zollhöfer, M.; Stamminger, M.; Theobalt, C.; Nießner, M. Face2Face: Real-time face capture and reenactment of RGB videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2387-2395, 2016.
Crossref
[16]
Fried, O.; Shechtman, E.; Goldman, D. B.; Finkelstein, A. Perspective-aware manipulation of portrait photos. ACM Transactions on Graphics Vol. 35, No. 4, Article No. 128, 2016.
Crossref Google Scholar
[17]
Shu, Z. X.; Hadap, S.; Shechtman, E.; Sunkavalli, K.; Paris, S.; Samaras, D. Portrait lighting transfer using a mass transport approach. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 2, 2017.
Crossref Google Scholar
[18]
Nagano, K.; Luo, H. W.; Wang, Z. J.; Seo, J.; Xing, J.; Hu, L. W.; Wei, L.; Li, H. Deep face normalization. ACM Transactions on Graphics Vol. 38, No. 6, Article No. 183, 2019.
Crossref Google Scholar
[19]
Zollhöfer, M.; Thies, J.; Garrido, P.; Bradley, D.; Beeler, T.; Pérez, P.; Stamminger, M.; Nießner, M.; Theobalt, C. State of the art on monocular 3D face reconstruction, tracking, and applications. Computer Graphics Forum Vol. 37, No. 2, 523-550, 2018.
Crossref Google Scholar
[20]
Blanz, V.; Vetter, T. A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, 187-194, 1999.
Crossref
[21]
Cao, C.; Weng, Y. L.; Zhou, S.; Tong, Y. Y.; Zhou, K. FaceWarehouse: A 3D facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics Vol. 20, No. 3, 413-425, 2014.
Crossref Google Scholar
[22]
Li, T. Y.; Bolkart, T.; Black, M. J.; Li, H.; Romero, J. Learning a model of facial shape and expression from 4D scans. ACM Transactions on Graphics Vol. 36, No. 6, Article No. 194, 2017.
Crossref Google Scholar
[23]
Jiang, L.; Zhang, J. Y.; Deng, B. L.; Li, H.; Liu, L. G. 3D face reconstruction with geometry details from a single image. IEEE Transactions on Image Processing Vol. 27, No. 10, 4756-4770, 2018.
Crossref Google Scholar
[24]
Richardson, E.; Sela, M. T.; Kimmel, R. 3D face reconstruction by learning from synthetic data. In: Proceedings of the 4th International Conference on 3D Vision, 460-469, 2016.
Crossref
[25]
Zhu, X. Y.; Lei, Z.; Liu, X. M.; Shi, H. L.; Li, S. Z. Face alignment across large poses: A 3D solution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 146-155, 2016.
Crossref
[26]
Richardson, E.; Sela, M. T.; Or-El, R.; Kimmel, R. Learning detailed face reconstruction from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5553-5562, 2017.
Crossref
[27]
Jackson, A. S.; Bulat, A.; Argyriou, V.; Tzimiropoulos, G. Large pose 3D face reconstruction from a single image via direct volumetric CNN regression. In: Proceedings of the IEEE International Conference on Computer Vision, 1031-1039, 2017.
Crossref
[28]
Sela, M. T.; Richardson, E.; Kimmel, R. Unrestricted facial geometry reconstruction using image-to-image translation. In: Proceedings of the IEEE International Conference on Computer Vision, 1585-1594, 2017.
Crossref
[29]
Tewari, A.; Zollhöfer, M.; Kim, H.; Garrido, P.; Bernard, F.; Pérez, P.; Theobalt, C. MoFA: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. In: Proceedings of the IEEE International Conference on Computer Vision, 3735-3744, 2017.
Crossref
[30]
Tran, L.; Liu, X. Nonlinear 3D face morphable model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7346-7355, 2018.
Crossref
[31]
Genova, K.; Cole, F.; Maschinot, A.; Sarna, A.; Vlasic, D.; Freeman, W. T. Unsupervised training for 3D morphable model regression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8377-8386, 2018.
Crossref
[32]
Gecer, B.; Ploumpis, S.; Kotsia, I.; Zafeiriou, S. GANFIT: Generative adversarial network fitting for high fidelity 3D face reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1155-1164, 2019.
Crossref
[33]
Guo, Y. D.; Zhang, J. Y.; Cai, J. F.; Jiang, B. Y.; Zheng, J. M. CNN-based real-time dense face reconstruction with inverse-rendered photo-realistic face images. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 41, No. 6, 1294-1307, 2019.
Crossref Google Scholar
[34]
Deng, Y.; Yang, J. L.; Xu, S. C.; Chen, D.; Jia, Y. D.; Tong, X. Accurate 3D face reconstruction with weakly-supervised learning: From single image to image set. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 285-295, 2019.
Crossref
[35]
Tewari, A.; Bernard, F.; Garrido, P.; Bharaj, G.; Elgharib, M.; Seidel, H. P.; Pérez, P.; Zollhöfer, M.; Theobalt, C. FML: Face model learning from videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10804-10814, 2019.
Crossref
[36]
Garrido, P.; Zollhöfer, M.; Casas, D.; Valgaerts, L.; Varanasi, K.; Pérez, P.; Theobalt, C. Reconstruction of personalized 3D face rigs from monocular video. ACM Transactions on Graphics Vol. 35, No. 3, Article No. 28, 2016.
Crossref Google Scholar
[37]
Cao, C.; Hou, Q. M.; Zhou, K. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Transactions on Graphics Vol. 33, No. 4, Article No. 43, 2014.
Crossref Google Scholar
[38]
Zhai, D. M.; Liu, X. M.; Ji, X. Y.; Zhao, D. B.; Gao, W. Joint gaze correction and face beautification for conference video using dual sparsity prior. IEEE Transactions on Industrial Electronics Vol. 66, No. 12, 9601-9611, 2019.
Crossref Google Scholar
[39]
Hassner, T.; Harel, S.; Paz, E.; Enbar, R. Effective face frontalization in unconstrained images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4295-4304, 2015.
Crossref
[40]
Zhao, Y. J.; Huang, Z.; Li, T. Y.; Chen, W. K.; Legendre, C., Ren, X. L.; Shapiro, A.; Li, H. Learning perspective undistortion of portraits. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 7848-7858, 2019.
Crossref
[41]
Yin, Y.; Jiang, S. Y.; Robinson, J. P.; Fu, Y. Dual-attention GAN for large-pose face frontalization. In: Proceedings of the 15th IEEE International Conference on Automatic Face and Gesture Recognition, 249-256, 2020.
Crossref
[42]
Paysan, P.; Knothe, R.; Amberg, B.; Romdhani, S.; Vetter, T. A 3D face model for pose and illumination invariant face recognition. In: Proceedings of the 6th IEEE International Conference on Advanced Video and Signal Based Surveillance, 296-301, 2009.
Crossref
[43]
Sumner, R. W.; Popović, J. Deformation transfer for triangle meshes. ACM Transactions on Graphics Vol. 23, No. 3, 399-405, 2004.
Crossref Google Scholar
[44]
Müller, C. Spherical Harmonics. Springer Berlin Heidelberg, 1966.
Crossref
[45]
He, K. M.; Zhang, X. Y.; Ren, S. Q.; Sun, J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770-778, 2016.
[46]
Bulat, A.; Tzimiropoulos, G. How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In: Proceedings of the IEEE International Conference on Computer Vision, 1021-1030, 2017.
Crossref
[47]
Burt, P. J.; Adelson, E. H. A multiresolution spline with application to image mosaics. ACM Transactions on Graphics Vol. 2, No. 4, 217-236, 1983.
Crossref Google Scholar
[48]
Kwatra, V.; Schödl, A.; Essa, I.; Turk, G.; Bobick, A. Graphcut textures. ACM Transactions on Graphics Vol. 22, No. 3, 277-286, 2003.
Crossref Google Scholar
[49]
Boykov, Y.; Kolmogorov, V. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 26, No. 9, 1124-1137, 2004.
Crossref Google Scholar
[50]
Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention. Lecture Notes in Computer Science, Vol. 9351. Navab, N.; Hornegger, J.; Wells, W.; Frangi, A. Eds. Springer Cham, 234-241, 2015.
Crossref
[51]
Bouaziz, S.; Deuss, M.; Schwartzburg, Y.; Weise, T.; Pauly, M. Shape-up: Shaping discrete geometry with projections. Computer Graphics Forum Vol. 31, No. 5, 1657-1667, 2012.
Crossref Google Scholar
[52]
Guillemot, C.; Le Meur, O. Image inpainting: Overview and recent advances. IEEE Signal Processing Magazine Vol. 31, No. 1, 127-144, 2014.
Crossref Google Scholar
[53]
Kim, H.; Zollhöfer, M.; Tewari, A.; Thies, J.; Richardt, C.; Theobalt, C. InverseFaceNet: Deep monocular inverse face rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4625-4634, 2018.
Crossref
[54]
Tewari, A.; Zollhöfer, M.; Garrido, P.; Bernard, F.; Kim, H.; Pérez, P.; Theobalt, C. Self-supervised multi-level face model learning for monocular reconstruction at over 250 Hz. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2549-2559, 2018.
Crossref
[55]
Tewari, A.; Bernard, F.; Garrido, P.; Bharaj, G.; Elgharib, M.; Seidel, H. P.; Pérez, P.; Zollhöfer, M.; Theobalt, C. FML: Face model learning from videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10804-10814, 2019.
Crossref
[56]
Schroff, F.; Kalenichenko, D.; Philbin, J. FaceNet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 815-823, 2015.
Crossref
Computational Visual Media
Volume 7 Issue 4,
December 2021
Pages 437-452
DOI: 10.1007/s41095-021-0215-y
Cite this article:
Guo Y, Zhang J, Chen Y, et al. Real-time face view correction for front-facing cameras. Computational Visual Media, 2021, 7(4): 437-452. https://doi.org/10.1007/s41095-021-0215-y

1105

Views

36

Downloads

0

Crossref

0

Web of Science

0

Scopus

0

CSCD

Altmetrics
Received: 29 January 2021
Accepted: 24 February 2021
Published: 27 April 2021
© The Author(s) 2021

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www. editorialmanager.com/cvmj.
Return