AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Computational Visual Media Article
PDF (17.9 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Research Article | Open Access

ShadowGAN: Shadow synthesis for virtual objects with conditional adversarial networks

University of Michigan, Ann Arbor, MI 48109 USA.
Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China.
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing 100191, China.
Show Author Information

Abstract

We introduce ShadowGAN, a generative adversarial network (GAN) for synthesizing shadows for virtual objects inserted in images. Given a target image containing several existing objects with shadows, and an input source object with a specified insertionposition, the network generates a realistic shadow for the source object. The shadow is synthesized by a generator; using the proposed local adversarial and global adversarial discriminators, the synthetic shadow’s appearance is locally realistic in shape, and globally consistent with other objects’ shadows in terms of shadow direction and area. To overcome the lack of training data, we produced training samples based on public 3D models and rendering technology. Experimental results from a user study show that the synthetic shadowed results look natural and authentic.

Keywords

shadow synthesisdeep learninggenera-tive adversarial networksimage synthesis

References

[1]
K. Karsch,; V. Hedau,; D. Forsyth,; D. Hoiem, Rendering synthetic objects into legacy photographs. ACM Transactions on Graphics Vol. 30, No. 6, Article No. 157, 2011.
Crossref Google Scholar
[2]
K. Karsch,; K. Sunkavalli,; S. Hadap,; N. Carr,; H. Jin,; R. Fonte,; M. Sittig,; D. Forsyth, Automatic scene inference for 3D object compositing. ACM Transactions on Graphics Vol. 33, No. 3, Article No. 32, 2014.
Crossref Google Scholar
[3]
E. Kee,; J. F. O’Brien,; H. Farid, Exposing photo manipulation from shading and shadows. ACM Transactions on Graphics Vol. 33, No. 5, Article No. 165, 2014.
Crossref Google Scholar
[4]
B. Liu,; K. Xu,; R. R. Martin, Static scene illumination estimation from videos with applications. Journal of Computer Science and Technology Vol. 32, No. 3, 430-442, 2017.
Crossref Google Scholar
[5]
S. Bell,; K. Bala,; N. Snavely, Intrinsic images in the wild. ACM Transactions on Graphics Vol. 33, No. 4, Article No. 159, 2014.
Crossref Google Scholar
[6]
S. Bi,; X. Han,; Y. Yu, An L1 image transform for edge-preserving smoothing and scene-level intrinsic decomposition. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 78, 2015.
Crossref Google Scholar
[7]
A. Bousseau,; S. Paris,; F. Durand, User-assisted intrinsic images. ACM Transactions on Graphics Vol. 28, No. 5, Article No. 130, 2009.
Crossref Google Scholar
[8]
Q. Fan,; J. Yang,; G. Hua,; B. Chen,; D. Wipf, Revisiting deep intrinsic image decompositions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8944-8952, 2018.
Crossref
[9]
A. Panagopoulos,; D. Samaras,; N. Paragios, Robust shadow and illumination estimation using a mixture model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 651-658, 2009.
Crossref
[10]
I. Sato,; Y. Sato,; K. Ikeuchi, Illumination from shadows. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 25, No. 3, 290-300, 2003.
Crossref Google Scholar
[11]
E. A. Khan,; E. Reinhard,; R. W. Fleming,; H. H. Bulthoff, Image-based material editing. ACM Transactions on Graphics Vol. 25, No. 3, 654-663, 2006.
Crossref Google Scholar
[12]
S. Ge,; X. Jin,; Q. Ye,; Z. Luo,; Q. Li, Image editing by object-aware optimal boundary searching and mixed-domain composition. Computational Visual Media Vol. 4, No. 1, 71-82, 2018.
Crossref Google Scholar
[13]
J. Kronander,; F. Banterle,; A. Gardner,; E. Miandji,; J. Unger, Photorealistic rendering of mixed reality scenes. Computer Graphics Forum Vol. 34, 643-665, 2015.
Crossref Google Scholar
[14]
P. Debevec, Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In: Proceedings of the ACM SIGGRAPH 2008 Classes, Article No. 32, 2008.
Crossref
[15]
Y. Shor,; D. Lischinski, The shadow meets the mask: Pyramid-based shadow removal. Computer Graphics Forum Vol. 27, No. 2, 577-586, 2008.
Crossref Google Scholar
[16]
J. Wang,; X. Li,; J. Yang, Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1788-1797, 2018.
Crossref
[17]
L. Qu,; J. Tian,; S. He,; Y. Tang,; R. W. H. Lau. DeshadowNet: A multi-context embedding deep network for shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4067-4075, 2017.
Crossref
[18]
L. Xu,; F. Qi,; R. Jiang, Shadow removal from a single image. In: Proceedings of the 6th International Conference on Intelligent Systems Design andApplications, 1049-1054, 2006.
Crossref
[19]
R. Guo,; Q. Dai,; D. Hoiem, Single-image shadow detection and removal using paired regions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2033-2040, 2011.
Crossref
[20]
L. Zhang,; Q. Zhang,; C. Xiao, Shadow remover: Image shadow removal based on illumination recovering optimization. IEEE Transactions on Image Processing Vol. 24, No. 11, 4623-4636, 2015.
Crossref Google Scholar
[21]
L.-Q. Ma,; J. Wang,; E. Shechtman,; K. Sunkavalli,; S.-M. Hu, Appearance harmonization for single image shadow removal. Computer Graphics Forum Vol. 35, No. 7, 189-197, 2016.
Crossref Google Scholar
[22]
I. Goodfellow,; J. Pouget-Abadie,; M. Mirza,; B. Xu,; D. Warde-Farley,; S. Ozair,; A. Courville,; Y. Bengio, Generative adversarial nets. In: Proceedings of the Advances in Neural Information Processing Systems 27, 2672-2680, 2014.
[23]
M. Li,; W. Zuo,; D. Zhang, Deep identity-aware transfer of facial attributes. arXiv preprint arXiv:1610.05586, 2016.
[24]
Y. Choi,; M. Choi,; M. Kim,; J.-W. Ha,; S. Kim,; J. Choo, StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8789-8797, 2018.
Crossref
[25]
W. Shen,; R. Liu, Learning residual images for face attribute manipulation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4030-4038, 2017.
Crossref
[26]
C. Ledig,; L. Theis,; F. Huszar,; J. Caballero,; A. Cunningham,; A. Acosta,; A. Aitken,; A. Tejani,; J. Totz,; Z. Wang,; W. Shi, Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4681-4690, 2017.
Crossref
[27]
D. Pathak,; P. Krahenbuhl,; J. Donahue,; T. Darrell,; A. A. Efros, Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2536-2544, 2016.
Crossref
[28]
S. Iizuka,; E. Simo-Serra,; H. Ishikawa, Globally and locally consistent image completion. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 107, 2017.
Crossref Google Scholar
[29]
M. Mirza,; S. Osindero, Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.
[30]
P. Isola,; J.-Y. Zhu,; T. Zhou,; A. A. Efros, Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1125-1134, 2017.
Crossref
[31]
J.-Y. Zhu,; T. Park,; P. Isola,; A. A. Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEEInternational Conference on Computer Vision, 2223-2232, 2017.
Crossref
[32]
T. Portenier,; Q. Hu,; A. Szabo,; S. A. Bigdeli,; P. Favaro,; M. Zwicker, Faceshop: Deep sketch-based face image editing. ACM Transactions on Graphics Vol. 37, No. 4, Article No. 99, 2018.
Crossref Google Scholar
[33]
Z. Wei,; Y. Sun,; J. Lin,; S. Liu, Learning adaptive receptive fields for deep image parsing networks. Computational Visual Media Vol. 4, No. 3, 231-244, 2018.
Crossref Google Scholar
[34]
A. X. Chang,; T. Funkhouser,; L. Guibas,; P. Hanrahan,; Q. Huang,; Z. Li,; S. Savarese,; M. Savva,; S. Song,; H. Su,; J. Xiao,; L. Yi,; F. Yu, ShapeNet: An information-rich 3D model repository. arXiv preprint arXiv:1512.03012, 2015.
[35]
E. P. Lafortune,; Y. D. Willems, Bi-directional path tracing. In: Proceedings of the 3rd International Conference on Computational Graphics and Visualiza-tion Techniques, 145-153, 1993.
[36]
A. Shrivastava,; T. Pfister,; O. Tuzel,; J. Susskind,; W. Wang,; R. Webb, Learning from simulated and unsupervised images through adversarial training. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2107-2116, 2017.
Crossref
[37]
F. Yu,; V. Koltun, Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122, 2015.
[38]
D. P. Kingma,; J. Ba, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Computational Visual Media
Volume 5 Issue 1,
March 2019
Pages 105-115
DOI: 10.1007/s41095-019-0136-1
Cite this article:
Zhang S, Liang R, Wang M. ShadowGAN: Shadow synthesis for virtual objects with conditional adversarial networks. Computational Visual Media, 2019, 5(1): 105-115. https://doi.org/10.1007/s41095-019-0136-1

757

Views

50

Downloads

48

Crossref

N/A

Web of Science

57

Scopus

0

CSCD

Altmetrics
Revised: 28 December 2018
Accepted: 05 February 2019
Published: 08 April 2019
© The Author(s) 2019

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from thecopyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
Return