Home Computational Visual Media Article
PDF (44.6 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript
Research Article | Open Access

Exploring contextual priors for real-world image super-resolution

Guangdong–Hong Kong–Macao Joint Laboratory of Human–Machine Intelligence-Synergy Systems, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China
University of Chinese Academy of Sciences, Beijing 100049, China
Shanghai AI Laboratory, Shanghai, China
Show Author Information

Graphical Abstract

View original image Download original image

Abstract

Real-world blind image super-resolution is a challenging problem due to the absence of target high resolution images for training. Inspired by the recent success of the single image generation based method SinGAN, we tackle this challenging problem with a refined model SR-SinGAN, which can learn to perform single real image super-resolution. Firstly, we empirically find that downsampled LR input with an appropriate size can improve the robustness of the generation model. Secondly, we introduce a global contextual prior to provide semantic information. This helps to remove distorted pixels and improve the output fidelity. Finally, we design an image gradient based local contextual prior to guide detail generation. It can alleviate generated artifacts in smooth areas while preserving rich details in densely textured regions (e.g., hair, grass). To evaluate the effectiveness of these contextual priors, we conducted extensive experiments on both artificial and real images. Results show that these priors can stabilize training and preserve output fidelity, improving the generated image quality. We furthermore find that these single image generation based methods work better for images with repeated textures compared to general images.

Keywords

unsupervised learningblind super-resolutionimage contextimage generation

References

[1]
Shaham, T. R.; Dekel, T.; Michaeli, T. SinGAN: Learning a generative model from a single natural image. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 4569–4579, 2019.
Crossref
[2]

Dong, C.; Loy, C. C.; He, K. M.; Tang, X. O. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 38, No. 2, 295–307, 2015.

Crossref Google Scholar
[3]
Zhang, Y. L.; Li, K. P.; Li, K.; Wang, L. C.; Zhong, B. N.; Fu, Y. Image super-resolution using very deep residual channel attention networks. In: Computer Vision – ECCV 2018. Lecture Notes in Computer Science, Vol. 11211. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 294–310, 2018.
Crossref
[4]
Zhang, K.; Zuo, W. M.; Zhang, L. Learning a single convolutional super-resolution network for multiple degradations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3262–3271, 2018.
Crossref
[5]
Gu, J. J.; Lu, H. N.; Zuo, W. M.; Dong, C. Blind super-resolution with iterative kernel correction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1604–1613, 2019.
Crossref
[6]
Ignatov, A.; Kobyshev, N.; Timofte, R.; Vanhoey, K.; Van Gool, L. WESPE: Weakly supervised photo enhancer for digital cameras. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 804–809, 2018.
Crossref
[7]
Lugmayr, A.; Danelljan, M.; Timofte, R. Unsupervised learning for real-world super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop, 3408–3416, 2019.
Crossref
[8]
Yuan, Y.; Liu, S. Y.; Zhang, J. W.; Zhang, Y. B.; Dong, C.; Lin, L. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 814–823, 2018.
Crossref
[9]
Shocher, A.; Cohen, N.; Irani, M. Zero-shot super-resolution using deep internal learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3118–3126, 2018.
Crossref
[10]
Lempitsky, V.; Vedaldi, A.; Ulyanov, D. Deep image prior. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9446–9454, 2018.
Crossref
[11]

Liu, A. R.; Liu, Y. H.; Gu, J. J.; Qiao, Y.; Dong, C. Blind image super-resolution: A survey and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 45, No. 5, 5461–5480, 2023.

Google Scholar
[12]
Glasner, D.; Bagon, S.; Irani, M. Super-resolution from a single image. In: Proceedings of the IEEE 12th International Conference on Computer Vision, 349–356, 2009.
Crossref
[13]
Zontak, M.; Irani, M. Internal statistics of a single natural image. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, 977–984, 2011.
Crossref
[14]
Michaeli, T.; Irani, M. Blind deblurring using internal patch recurrence. In: Computer Vision – ECCV 2014. Lecture Notes in Computer Science, Vol. 8691. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer Cham, 783–798, 2014.
Crossref
[15]

Kim, K. I.; Kwon, Y. Single-image super-resolution using sparse regression and natural image prior. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 32, No. 6, 1127–1133, 2010.

Crossref Google Scholar
[16]

Yang, W. M.; Zhang, X. C.; Tian, Y. P.; Wang, W.; Xue, J. H.; Liao, Q. M. Deep learning for single image super-resolution: A brief review. IEEE Transactions on Multimedia Vol. 21, No. 12, 3106–3121, 2019.

Crossref Google Scholar
[17]

Liu, S.; Gang, R. P.; Li, C. H.; Song, R. X. Adaptive deep residual network for single image super-resolution. Computational Visual Media Vol. 5, No. 4, 391–401, 2019.

Crossref Google Scholar
[18]
Ledig, C.; Theis, L.; Huszár, F.; Caballero, J.; Cunningham, A.; Acosta, A.; Aitken, A.; Tejani, A.; Totz, J.; Wang, Z. H.; et al. Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 105–114, 2017.
Crossref
[19]
Goodfellow, I. J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, Vol. 2, 2672–2680, 2014.
[20]
Bhat, G.; Danelljan, M.; Van Gool, L.; Timofte, R. Deep burst super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9205–9214, 2021.
Crossref
[21]
Zhang, K.; Liang, J. Y.; Van Gool, L.; Timofte, R. Designing a practical degradation model for deep blind image super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 4771–4780, 2021.
Crossref
[22]
Wang, X. T.; Xie, L. B.; Dong, C.; Shan, Y. Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 1905–1914, 2021.
Crossref
[23]
Begin, I.; Ferrie, F. R. Blind super-resolution using a learning-based approach. In: Proceedings of the 17th International Conference on Pattern Recognition, 85–89, 2004.
Crossref
[24]
Michaeli, T.; Irani, M. Nonparametric blind super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision, 945–952, 2013.
Crossref
[25]
Bulat, A.; Yang, J.; Tzimiropoulos, G. To learn image super-resolution, use a GAN to learn how to do image degradation first. In: Computer Vision – ECCV 2018. Lecture Notes in Computer Science, Vol. 11210. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 187–202, 2018.
Crossref
[26]
Zhang, X. E.; Chen, Q. F.; Ng, R.; Koltun, V. Zoom to learn, learn to zoom. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3757–3765, 2019.
Crossref
[27]

Farsiu, S.; Robinson, M. D.; Elad, M.; Milanfar, P. Fast and robust multiframe super resolution. IEEE Transactions on Image Processing Vol. 13, No. 10, 1327–1344, 2004.

Crossref Google Scholar
[28]
Cai, J.; Gu, S.; Timofte, R.; Zhang, L.; Liu, X.; Ding, Y.; He, D.; Li, C.; Fu, Y.; Wen, S.; et al. NTIRE 2019 challenge on real image super-resolution: Methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2211–2223, 2019.
[29]
Liang, J. Y.; Zhang, K.; Gu, S. H.; Van Gool, L.; Timofte, R. Flow-based kernel prior with application to blind super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10596–10605, 2021.
Crossref
[30]
Bell-Kligler, S.; Shocher, A.; Irani, M. Blind super-resolution kernel estimation using an internal-GAN. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems, Article No. 26, 284–293, 2019.
[31]

Freedman, G.; Fattal, R. Image and video upscaling from local self-examples. ACM Transactions on Graphics Vol. 30, No. 2, Article No. 12, 2011.

Crossref Google Scholar
[32]
Yang, C. Y.; Yang, M. H. Fast direct super-resolution by simple functions. In: Proceedings of the IEEE International Conference on Computer Vision, 561–568, 2013.
Crossref
[33]

Zhang, K.; Zuo, W. M.; Chen, Y. J.; Meng, D. Y.; Zhang, L. Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing Vol. 26, No. 7, 3142–3155, 2017.

Crossref Google Scholar
[34]
Gandelsman, Y.; Shocher, A.; Irani, M. “double-DIP”: Unsupervised image decomposition via coupled deep-image-priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11018–11027, 2019.
Crossref
[35]
Ma, C.; Rao, Y. M.; Cheng, Y. A.; Chen, C.; Lu, J. W.; Zhou, J. Structure-preserving super resolution with gradient guidance. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7766–7775, 2020.
Crossref
[36]
Le, V.; Brandt, J.; Lin, Z.; Bourdev, L.; Huang, T. S. Interactive facial feature localization. In: Computer Vision – ECCV 2012. Lecture Notes in Computer Science, Vol. 7574. Fitzgibbon, A.; Lazebnik, S.; Perona, P.; Sato, Y.; Schmid, C. Eds. Springer Berlin Heidelberg, 679–692, 2012.
Crossref
[37]
Liu, Z. W.; Luo, P.; Wang, X. G.; Tang, X. O. Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision, 3730–3738, 2015.
Crossref
[38]
Lugmayr, A.; Danelljan, M.; Timofte, R.; Ahn, N.; Bai, D.; Cai, J.; Cao, Y.; Chen, J. Y.; Cheng, K. H.; Chun, S.; et al. NTIRE 2020 challenge on real-world image super-resolution: Methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2058–2076, 2020.
[39]
Zhang, R.; Isola, P.; Efros, A. A.; Shechtman, E.; Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 586–595, 2018.
Crossref
[40]

Wang, Z.; Bovik, A. C.; Sheikh, H. R.; Simoncelli, E. P. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing Vol. 13, No. 4, 600–612, 2004.

Crossref Google Scholar
[41]
Blau, Y.; Mechrez, R.; Timofte, R.; Michaeli, T.; Zelnik-Manor, L. The 2018 PIRM challenge on perceptual image super-resolution. In: Computer Vision – ECCV 2018 Workshops. Lecture Notes in Computer Science, Vol. 11133. Leal-Taixé, L.; Roth, S. Eds. Springer Cham, 334–355, 2019.
Crossref
[42]

Mittal, A.; Soundararajan, R.; Bovik, A. C. Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters Vol. 20, No. 3, 209–212, 2013.

Crossref Google Scholar
[43]

Ma, C.; Yang, C. Y.; Yang, X. K.; Yang, M. H. Learning a no-reference quality metric for single-image super-resolution. Computer Vision and Image Understanding Vol. 158, 1–16, 2017.

Crossref Google Scholar
[44]
Kingma, D. P.; Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[45]
Gu, J. J.; Dong, C. Interpreting super-resolution networks with local attribution maps. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9195–9204, 2021.
Crossref
[46]

Lebrun, M.; Colom, M.; Morel, J. M. The noise clinic: A blind image denoising algorithm. Image Processing on Line Vol. 5, 1–54, 2015.

Crossref Google Scholar
Computational Visual Media
Volume 11 Issue 1,
February 2025
Pages 159-177
DOI: 10.26599/CVM.2025.9450303
Cite this article:
Wu S, Dong C, Qiao Y. Exploring contextual priors for real-world image super-resolution. Computational Visual Media, 2025, 11(1): 159-177. https://doi.org/10.26599/CVM.2025.9450303
Metrics & Citations  
Article History
Copyright
Rights and Permissions
Return