AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Computational Visual Media Article
PDF (12 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Research Article | Open Access

Practical automatic background substitution for live video

Department of Computer Science, Tsinghua University, Beijing, 100084, China.
School of Computer Science and Informatics, Cardiff University, Cardiff, CF24 3AA, UK.
Show Author Information

Abstract

In this paper we present a novel automatic background substitution approach for live video. The objective of background substitution is to extract the foreground from the input video and then combine it with a new background. In this paper, we use a color line model to improve the Gaussian mixture model in the background cut method to obtain a binary foreground segmentation result that is less sensitive to brightness differences. Based on the high quality binary segmentation results, we can automatically create a reliable trimap for alpha matting to refine the segmentation boundary. To make the composition result more realistic, an automatic foreground color adjustment step is added to make the foreground look consistent with the new background. Compared to previous approaches, our method can produce higher quality binary segmentation results, and to the best of our knowledge, this is the first time such an automatic and integrated background substitution system has been proposed which can run in real time, which makes it practical for everyday applications.

Keywords

background substitutionbackground replacementbackground subtractionalpha matting

Electronic Supplementary Material

Video
41095_2016_74_MOESM1_ESM.mp4

References

[1]
X. Bai,; J. Wang,; D. Simons,; G. Sapiro, Video SnapCut: Robust video object cutout using localized classifiers. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 70, 2009.
Crossref Google Scholar
[2]
T. Chen,; J.-Y. Zhu,; A. Shamir,; S.-M. Hu, Motion-aware gradient domain video composition. IEEE Transactions on Image Processing Vol. 22, No. 7, 2532-2544, 2013.
Crossref Google Scholar
[3]
Z. Liu,; M. Cohen, Head-size equalization for better visual perception of video conferencing. In: Proceedings of the IEEE International Conference on Multimedia and Expo, 4, 2005.
[4]
Z. Zhu,; R. R. Martin,; R. Pepperell,; A. Burleigh, 3D modeling and motion parallax for improved videoconferencing. Computational Visual Media Vol. 2, No. 2, 131-142, 2016.
Crossref Google Scholar
[5]
D. W. F. Van Krevelen,; R. Poelman, A survey of augmented reality technologies, applications and limitations. International Journal of Virtual Reality Vol. 9, No. 2, 1-21, 2010.
Crossref Google Scholar
[6]
N. Apostoloff,; A. Fitzgibbon, Bayesian video matting using learnt image priors. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1, I-407-I-414, 2004.
[7]
T. Bouwmans,; F. El Baf,; B. Vachon, Background modeling using mixture of Gaussians for foreground detection—A survey. Recent Patents on Computer Science Vol. 1, No. 3, 219-237, 2008.
Crossref Google Scholar
[8]
L. Maddalena,; A. Petrosino, A self-organizing approach to background subtraction for visual surveillance applications. IEEE Transactions on Image Processing Vol. 17, No. 7, 1168-1177, 2008.
Crossref Google Scholar
[9]
D.-M. Tsai,; S.-C. Lai, Independent component analysis-based background subtraction for indoor surveillance. IEEE Transactions on Image Processing Vol. 18, No. 1, 158-167, 2009.
Crossref Google Scholar
[10]
O. Barnich,; M. Van Droogenbroeck, ViBe: A universal background subtraction algorithm for video sequences. IEEE Transactions on Image Processing Vol. 20, No. 6, 1709-1724, 2011.
Crossref Google Scholar
[11]
M. Hofmann,; P. Tiefenbacher,; G. Rigoll, Background segmentation with feedback: The pixel-based adaptive segmenter. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 38-43, 2012.
Crossref
[12]
J. Sun,; W. Zhang,; X. Tang,; H.-Y. Shum, Background cut. In: Computer Vision-ECCV 2006. A. Leonardis,; H. Bischof,; A. Pinz, Eds. Springer Berlin Heidelberg, 628-641, 2006.
Crossref
[13]
A. Criminisi,; G. Cross,; A. Blake,; V. Kolmogorov, Bilayer segmentation of live video. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 53-60, 2006.
[14]
P. Yin,; A. Criminisi,; J. Winn,; I. Essa, Bilayer segmentation of webcam videos using tree-based classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 33, No. 1, 30-42, 2011.
Crossref Google Scholar
[15]
Y.-Y. Chuang,; A. Agarwala,; B. Curless,; D. H. Salesin,; R. Szeliski, Video matting of complex scenes. ACM Transactions on Graphics Vol. 21, No. 3, 243-248, 2002.
Crossref Google Scholar
[16]
M. Gong,; Y. Qian,; L. Cheng, Integrated foreground segmentation and boundary matting for live videos. IEEE Transactions on Image Processing Vol. 24, No. 4, 1356-1370, 2015.
Crossref Google Scholar
[17]
P. Pérez,; M. Gangnet,; A. Blake, Poisson image editing. ACM Transactions on Graphics Vol. 22, No. 3, 313-318, 2003.
Crossref Google Scholar
[18]
J. Jia,; J. Sun,; C.-K. Tang,; H.-Y. Shum, Drag-and-drop pasting. ACM Transactions on Graphics Vol. 25, No. 3, 631-637, 2006.
Crossref Google Scholar
[19]
G. Buchsbaum, A spatial processor model for object colour perception. Journal of the Franklin Institute Vol. 310, No. 1, 1-26, 1980.
Crossref Google Scholar
[20]
G. D. Finlayson,; S. D. Hordley,; P. M. Hubel, Color by correlation: A simple, unifying framework for color constancy. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 23, No. 11, 1209-1221, 2001.
Crossref Google Scholar
[21]
D. Cheng,; D. K. Prasad,; M. S. Brown, Illuminant estimation for color constancy: Why spatial-domain methods work and the role of the color distribution. Journal of the Optical Society of America A Vol. 31, No. 5, 1049-1058, 2014.
Crossref Google Scholar
[22]
D. Cheng,; B. Price,; S. Cohen,; M. S. Brown, Beyond white: Ground truth colors for color constancy correction. In: Proceedings of the IEEE International Conference on Computer Vision, 298-306, 2015.
Crossref
[23]
I. Omer,; M. Werman, Color lines: Image specific color representation. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2, II-946-II-953, 2004.
[24]
A. Levin,; D. Lischinski,; Y. Weiss, A closed-form solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 30, No. 2, 228-242, 2008.
Crossref Google Scholar
[25]
E. H. Land,; J. J. McCann, Lightness and retinex theory. Journal of the Optical Society of America Vol. 61, No. 1, 1-11, 1971.
Crossref Google Scholar
[26]
Y. Zhang,; Y.-L. Tang,; K.-L. Cheng, Efficient video cutout by paint selection. Journal of Computer Science and Technology Vol. 30, No. 3, 467-477, 2015.
Crossref Google Scholar
[27]
A. R. Smith,; J. F. Blinn, Blue screen matting. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, 259-268, 1996.
Crossref
[28]
A. Mumtaz,; W. Zhang,; A. B. Chan, Joint motion segmentation and background estimation in dynamic scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 368-375, 2014.
Crossref
[29]
L. Zhang,; H. Huang,; H. Fu, EXCOL: An extract-and-complete layering approach to cartoon animation reusing. IEEE Transactions on Visualization and Computer Graphics Vol. 18, No. 7, 1156-1169, 2012.
Crossref Google Scholar
[30]
E. S. L. Gastal,; M. M. Oliveira, Shared sampling for real-time alpha matting. Computer Graphics Forum Vol. 29, No. 2, 575-584, 2010.
Crossref Google Scholar
[31]
X. Chen,; D. Zou,; S. Zhou,; Q. Zhao,; P. Tan, Image matting with local and nonlocal smooth priors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1902-1907, 2013.
Crossref
[32]
B.-Y. Wong,; K.-T. Shih,; C.-K. Liang,; H. H. Chen, Single image realism assessment and recoloring by color compatibility. IEEE Transactions on Multimedia Vol. 14, No. 3, 760-769, 2012.
Crossref Google Scholar
[33]
D. Cohen-Or,; O. Sorkine,; R. Gal,; T. Leyvand,; Y.-Q. Xu, Color harmonization. ACM Transactions on Graphics Vol. 25, No. 3, 624-630, 2006.
Crossref Google Scholar
[34]
Z. Kuang,; P. Lu,; X. Wang,; X. Lu, Learning self-adaptive color harmony model for aesthetic quality classification. In: Proceedings of SPIE 9443, the 6th International Conference on Graphic and Image Processing, 94431O, 2015.
Crossref
[35]
T. Chen,; M.-M. Cheng,; P. Tan,; A. Shamir,; S.-M. Hu, Sketch2Photo: Internet image montage. ACM Transactions on Graphics Vol. 28, No. 5, Article No. 124, 2009.
Crossref Google Scholar
[36]
Z. Farbman,; G. Hoffer,; Y. Lipman,; D. Cohen-Or,; D. Lischinski, Coordinates for instant image cloning. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 67, 2009.
Crossref Google Scholar
[37]
Y. Boykov,; V. Kolmogorov, An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. In: Energy Minimization Methods in Computer Vision and Pattern Recognition. M. Figueiredo,; J. Zerubia,; A. K. Jain, Eds. Springer Berlin Heidelberg, 359-374, 2001.
Crossref
[38]
M. H. Sigari,; N. Mozayani,; H. R. Pourreza, Fuzzy running average and fuzzy background subtraction: concepts and application. International Journal of Computer Science and Network Security Vol. 8, No. 2, 138-143, 2008.
Google Scholar
[39]
A. Sobral, BGSLibrary. 2016. Available at https://github.com/andrewssobral/bgslibrary.
[40]
P. L. Rosin,; E. Ioannidis, Evaluation of global image thresholding for change detection. Pattern Recognition Letters Vol. 24, No. 14, 2345-2356, 2003.
Crossref Google Scholar
[41]
D. J. Kerbyson,; T. J. Atherton, Circle detection using Hough transform filters. In: Proceedings of the 5th International Conference on Image Processing and its Applications, 370-374, 1995.
Crossref
[42]
Graphics and Media Lab. Videomatting benchmark. 2016. Available at http://videomatting.com.
Computational Visual Media
Volume 3 Issue 3,
September 2017
Pages 273-284
DOI: 10.1007/s41095-016-0074-0
Cite this article:
Huang H, Fang X, Ye Y, et al. Practical automatic background substitution for live video. Computational Visual Media, 2017, 3(3): 273-284. https://doi.org/10.1007/s41095-016-0074-0

823

Views

14

Downloads

15

Crossref

N/A

Web of Science

16

Scopus

2

CSCD

Altmetrics
Revised: 18 August 2016
Accepted: 20 December 2016
Published: 15 March 2017
© The Author(s) 2016

This article is published with open access at Springerlink.com

The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
Return