AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Computational Visual Media Article
PDF (9.8 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Research Article | Open Access

Pyramid-VAE-GAN: Transferring hierarchical latent variables for image inpainting

College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
Advanced Technology Research Institute, Zhejiang University, Hangzhou 310027, China
Show Author Information

Graphical Abstract

Abstract

Significant progress has been made in image inpainting methods in recent years. However, they are incapable of producing inpainting results with reasonable structures, rich detail, and sharpness at the same time. In this paper, we propose the Pyramid-VAE-GAN network for image inpainting to address this limitation. Our network is built on a variational autoencoder (VAE) backbone that encodes high-level latent variables to represent complicated high-dimensional prior distributions of images. The prior assists in reconstructing reasonable structures when inpainting. We also adopt a pyramid structure in our model to maintain rich detail in low-level latent variables. To avoid the usual incompatibility of requiring both reasonable structures and rich detail, we propose a novel cross-layer latent variable transfer module. This transfers information about long-range structures contained in high-level latent variables to low-level latent variables representing more detailed information. We further use adversarial training to select the most reasonable results and to improve the sharpness of the images. Extensive experimental results on multiple datasets demonstrate the superiority of our method. Our code is available at https://github.com/thy960112/Pyramid-VAE-GAN.

Keywords

image inpaintingvariational autoencoder (VAE)generative modellatent variable transfer (LTN);pyramid structure

References

[1]
Bertalmio, M.; Sapiro, G.; Caselles, V.; Ballester, C. Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, 417–424, 2000.
Crossref
[2]
Wang, N.; Zhang, Y. P.; Zhang, L. F. Dynamic selection network for image inpainting. IEEE Transactions on Image Processing Vol. 30, 1784–1798, 2021.
Crossref Google Scholar
[3]
Li, J. Y.; Wang, N.; Zhang, L. F.; Du, B.; Tao, D. C. Recurrent feature reasoning for image inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7757–7765, 2020.
Crossref
[4]
Wan, Z. Y.; Zhang, J. B.; Chen, D. D.; Liao, J. High-fidelity pluralistic image completion with transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 4672–4681, 2021.
Crossref
[5]
Lu, M.; Niu, S. Z. A detection approach using LSTM-CNN for object removal caused by exemplar-based image inpainting. Electronics Vol. 9, No. 5, 858, 2020.
Crossref Google Scholar
[6]
Shetty, R.; Fritz, M.; Schiele, B. Adversarial scene editing: Automatic object removal from weak supervision. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, 7717–7727, 2018.
[7]
Barnes, C.; Shechtman, E.; Finkelstein, A.; Goldman, D. B. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 24, 2009.
Crossref Google Scholar
[8]
Pathak, D.; Krähenbühl, P.; Donahue, J.; Darrell, T.; Efros, A. A. Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2536–2544, 2016.
Crossref
[9]
Yu, J. H.; Lin, Z.; Yang, J. M.; Shen, X. H.; Lu, X.; Huang, T. S. Generative image inpainting with contextual attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5505–5514, 2018.
Crossref
[10]
Wu, X.; Xu, K.; Hall, P. A survey of image synthesis and editing with generative adversarial networks. Tsinghua Science and Technology Vol. 22, No. 6, 660–674, 2017.
Crossref Google Scholar
[11]
Xue, Y.; Guo, Y. C.; Zhang, H.; Xu, T.; Zhang, S. H.; Huang, X. L. Deep image synthesis from intuitive user input: A review and perspectives. Computational Visual Media Vol. 8, No. 1, 3–31, 2022.
Crossref Google Scholar
[12]
Zeng, X. X.; Wu, Z. L.; Peng, X. J.; Qiao, Y. Joint 3D facial shape reconstruction and texture completion from a single image. Computational Visual Media Vol. 8, No. 2, 239–256, 2022.
Crossref Google Scholar
[13]
Wu, X.; Li, R. L.; Zhang, F. L.; Liu, J. C.; Wang, J.; Shamir, A.; Hu, S. M. Deep portrait image completion and extrapolation. IEEE Transactions on Image Processing Vol. 29, 2344–2355, 2020.
Crossref Google Scholar
[14]
Liu, H. Y.; Wan, Z. Y.; Huang, W.; Song, Y. B.; Han, X. T.; Liao, J. PD-GAN: Probabilistic diverse GAN for image inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9367–9376, 2021.
Crossref
[15]
Chen, Y. T.; Zhang, H. P.; Liu, L. W.; Chen, X.; Zhang, Q.; Yang, K.; Xia, R. L.; Xie, J. B. Research on image inpainting algorithm of improved GAN based on two-discriminations networks. Applied Intelligence Vol. 51, No. 6, 3460–3474, 2021.
Crossref Google Scholar
[16]
Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial networks. Communications of the ACM Vol. 63, No. 11, 139–144, 2020.
Crossref Google Scholar
[17]
Zeng, Y. H.; Fu, J. L.; Chao, H. Y.; Guo, B. N. Learning pyramid-context encoder network for high-quality image inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1486–1494, 2019.
Crossref
[18]
Kingma, D. P.; Welling, M. Auto-encoding variational bayes. In: Proceedings of the International Conference on Learning Representations, 2014.
[19]
Karras, T.; Aila, T. M.; Laine, S.; Lehtinen, J. Progressive growing of GANs for improved quality, stability, and variation. In: Proceedings of the International Conference on Learning Representations, 2018.
[20]
Krause, J.; Stark, M.; Jia, D.; Li, F. F. 3D object representations for fine-grained categorization. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, 554–561, 2013.
Crossref
[21]
Cimpoi, M.; Maji, S.; Kokkinos, I.; Mohamed, S.; Vedaldi, A. Describing textures in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3606–3613, 2014.
Crossref
[22]
Tyleček, R.; Šára, R. Spatial pattern templates for recognition of objects with regular structure. In: Pattern Recognition. GCPR 2013. Lecture Notes in Computer Science, Vol. 8142. Weickert, J.; Hein, M.; Schiele, B. Eds. Springer Berlin Heidelberg, 364–374, 2013.
Crossref
[23]
Barnes, C.; Zhang, F. L. A survey of the state-of-the-art in patch-based synthesis. Computational Visual Media Vol. 3, No. 1, 3–20, 2017.
Crossref Google Scholar
[24]
Fukushima, K.; Miyake, S. Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition. In: Competition and Cooperation in Neural Nets. Lecture Notes in Biomathematics, Vol. 45. Amari, S.; Arbib, M. A. Eds. Springer Berlin Heidelberg, 267–285, 1982.
Crossref
[25]
LeCun, Y.; Boser, B.; Denker, J. S.; Henderson, D.; Howard, R. E.; Hubbard, W.; Jackel, L. D. Backpropagation applied to handwritten zip code recognition. Neural Computation Vol. 1, No. 4, 541–551, 1989.
Crossref Google Scholar
[26]
Peng, J. L.; Liu, D.; Xu, S. C.; Li, H. Q. Generating diverse structure for image inpainting with hierarchical VQ-VAE. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10770–10779, 2021.
Crossref
[27]
Vahdat, A; Kautz, J. NVAE: A deep hierarchical variational autoencoder. In: Proceedings of the 34th International Conference on Neural Information Processing Systems, Article No. 1650, 19667–19679, 2020.
[28]
Ramesh, A.; Pavlov, M.; Goh, G.; Gray, S.; Voss, C.; Radford, A.; Chen, M.; Sutskever, I. Zero-shot text-to-image generation. In: Proceedings of the 38th International Conference on Machine Learning, Vol. 139, 8821–8831, 2021.
[29]
Bowman, S. R.; Vilnis, L.; Vinyals, O.; Dai, A. M.; Jozefowicz, R.; Bengio, S. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349.2015.
Google Scholar
[30]
Frazer, J.; Notin, P.; Dias, M.; Gomez, A.; Min, J. K.; Brock, K.; Gal, Y.; Marks, D. S. Disease variant prediction with deep generative models of evolutionary data. Nature Vol. 599, No. 7883, 91–95, 2021.
Crossref Google Scholar
[31]
Salimans, T.; Kingma, D. P.; Welling, M. Markov Chain Monte Carlo and variational inference: Bridging the gap. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning, Vol. 37, 1218–1226, 2015.
[32]
Rezende, D. J.; Mohamed, S.; Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. In: Proceedings of the 31st International Conference on International Conference on Machine Learning, Vol. 32, II-1278–II-1286, 2014.
[33]
Kulkarni, T. D.; Whitney, W. F.; Kohli, P.; Tenenbaum, J. B. Deep convolutional inverse graphics network. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, 2539–2547, 2015.
[34]
Sun, R. Q.; Huang, C.; Zhu, H. L.; Ma, L. Z. Mask-aware photorealistic facial attribute manipulation. Computational Visual Media Vol. 7, No. 3, 363–374, 2021.
Crossref Google Scholar
[35]
Walker, J.; Doersch, C.; Gupta, A.; Hebert, M. An uncertain future: Forecasting from static images using variational autoencoders. In: Computer Vision – ECCV 2016. Lecture Notes in Computer Science, Vol. 9911. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 835–851, 2016.
Crossref
[36]
Sohn, K.; Yan, X. C.; Lee, H. Learning structured output representation using deep conditional generative models. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, Vol. 2, 3483–3491, 2015.
[37]
Gao, R.; Hou, X. S.; Qin, J.; Chen, J. X.; Liu, L.; Zhu, F.; Zhang, Z.; Shao, L. Zero-VAE-GAN: Generating unseen features for generalized and transductive zero-shot learning. IEEE Transactions on Image Processing Vol. 29, 3665–3680, 2020.
Crossref Google Scholar
[38]
Zheng, C. X.; Cham, T. J.; Cai, J. F. Pluralistic image completion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1438–1447, 2019.
Crossref
[39]
Gonzalez, R. C.; Woods, R. E. Digital Image Processing, 4th edn. Pearson, 2018.
[40]
Lim, J. H.; Ye, J. C. Geometric GAN. arXiv preprint arXiv:1705.02894, 2017.
Google Scholar
[41]
Fu, M. C. Stochastic gradient estimation. In: Handbook of Simulation Optimization. International Series in Operations Research & Management Science, Vol. 216. Fu, M. Ed. Springer New York, 105–147, 2015.
Crossref
[42]
Devroye, L. Sample-based non-uniform random variate generation. In: Proceedings of the 18th Conference on Winter Simulation, 260–265, 1986.
Crossref
[43]
Doersch, C. Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908, 2016.
Google Scholar
[44]
Iizuka, S.; Simo-Serra, E.; Ishikawa, H. Globally and locally consistent image completion. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 107, 2017.
Crossref Google Scholar
[45]
Liu, G. L.; Reda, F. A.; Shih, K. J.; Wang, T. C.; Tao, A.; Catanzaro, B. Image inpainting for irregular holes using partial convolutions. In: Computer Vision –ECCV 2018. Lecture Notes in Computer Science, Vol. 11215. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 89–105, 2018.
Crossref
[46]
Yu, J. H.; Lin, Z.; Yang, J. M.; Shen, X. H.; Lu, X.; Huang, T. Free-form image inpainting with gated convolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 4470–4479, 2019.
Crossref
[47]
Cortes, C.; Vapnik, V. Support-vector networks. Machine Learning Vol. 20, No. 3, 273–297, 1995.
Crossref Google Scholar
[48]
Wang, Z.; Simoncelli, E. P.; Bovik, A. C. Multiscale structural similarity for image quality assessment. In: Proceedings of the 37th Asilomar Conference on Signals, Systems & Computers, 1398–1402, 2003.
[49]
Szeliski, R. Computer Vision: Algorithms and Applications. Springer London, 2011.
Crossref
[50]
Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; Hochreiter, S. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, 6629–6640, 2017.
Computational Visual Media
Volume 9 Issue 4,
December 2023
Pages 827-841
DOI: 10.1007/s41095-022-0331-3
Cite this article:
Tian H, Zhang L, Li S, et al. Pyramid-VAE-GAN: Transferring hierarchical latent variables for image inpainting. Computational Visual Media, 2023, 9(4): 827-841. https://doi.org/10.1007/s41095-022-0331-3

503

Views

15

Downloads

3

Crossref

4

Web of Science

4

Scopus

0

CSCD

Altmetrics
Received: 02 May 2022
Accepted: 18 December 2022
Published: 27 July 2023
© The Author(s) 2023.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
Return