AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Computational Visual Media Article
PDF (7.6 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Research Article | Open Access

Light field super-resolution using complementary-view feature attention

Faculty of Applied Sciences, Macao Polytechnic University, Macao SAR 999078, China
State Key Laboratory of Software Development Environment, School of Computer Science and Engineering, Beihang University, Beijing 100191, China, and Beihang Hangzhou Innovation Institute Yuhang, Xixi Octagon City, Yuhang District, Hangzhou 310023, China
Show Author Information

Graphical Abstract

Abstract

Light field (LF) cameras record multiple perspectives by a sparse sampling of real scenes, and these perspectives provide complementary information. This information is beneficial to LF super-resolution (LFSR). Compared with traditional single-imagesuper-resolution, LF can exploit parallax structure and perspective correlation among different LF views. Furthermore, the performance of existing methods are limited as they fail to deeply explore the complementary information across LF views. In this paper, we propose a novel network, called the light field complementary-view feature attention network (LF-CFANet), to improve LFSR by dynamically learning the complementary information in LF views. Specifically, we design a residual complementary-view spatial and channel attention module (RCSCAM) to effectively interact with complementary information between complementary views. Moreover, RCSCAM captures the relationships between different channels, and it is able to generate informative features for reconstructing LF images while ignoring redundant information. Then, a maximum-difference information supplementary branch (MDISB) is used to supplement information from the maximum-difference angular positions based on the geometric structure of LF images. This branch also can guide the process of reconstruction. Experimental results on both synthetic and real-world datasets demonstrate the superiority of our method. The proposed LF-CFANet has a more advanced reconstruction performance that displays faithful details with higher SR accuracy than state-of-the-art methods.

Keywords

light field (LF)super-resolution (SR)attention

References

[1]
Huang, F.-C.; Luebke, D.; Wetzstein, G. The light field stereoscope. In: Proceedings of the ACM SIGGRAPH Emerging Technologies, Article No. 24, 2015.
Crossref
[2]
Yu, J. A light-field journey to virtual reality. IEEE MultiMedia Vol. 24, No. 2, 104–112, 2017.
Crossref Google Scholar
[3]
Sheng, H.; Wang, S.; Zhang, Y.; Yu, D. X.; Cheng, X. Z.; Lyu, W. F.; Xiong, Z. Near-online tracking with co-occurrence constraints in blockchain-based edge computing. IEEE Internet of Things Journal Vol. 8, No. 4, 2193–2207, 2021.
Crossref Google Scholar
[4]
Sheng, H.; Zhang, Y.; Wu, Y. B.; Wang, S.; Lyu, W. F.; Ke, W.; Xiong, Z. Hypothesis testing based tracking with spatio-temporal joint interaction modeling. IEEE Transactions on Circuits and Systems for Video Technology Vol. 30, No. 9, 2971–2983, 2020.
Crossref Google Scholar
[5]
Wang, S.; Sheng, H.; Zhang, Y.; Wu, Y. B.; Xiong, Z. A general recurrent tracking framework without real data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 13199–13208, 2021.
Crossref
[6]
Wang, S.; Sheng, H.; Zhang, Y.; Yang, D.; Shen, J.; Chen, R. Blockchain-empowered distributed multi-camera multi-target tracking in edge computing. IEEE Transactions on Industrial Informatics, , 2023.
Crossref Google Scholar
[7]
Zhu, H.; Wang, Q.; Yu, J. Y. Occlusion-model guided antiocclusion depth estimation in light field. IEEE Journal of Selected Topics in Signal Processing Vol. 11, No. 7, 965–978, 2017.
Crossref Google Scholar
[8]
Piao, Y. R.; Li, X.; Zhang, M.; Yu, J. Y.; Lu, H. C. Saliency detection via depth-induced cellular automata on light field. IEEE Transactions on Image Processing Vol. 29, 1879–1889, 2020.
Crossref Google Scholar
[9]
Zhang, M.; Li, J.; Ji, W.; Piao, Y.; Lu, H. Memory-oriented decoder for light field salient object detection. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems, Article No. 81, 898–908, 2019.
[10]
Bishop, T. E.; Favaro, P. The light field camera: Extended depth of field, aliasing, and superresolution. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 34, No. 5, 972–986, 2012.
Crossref Google Scholar
[11]
Wanner, S.; Goldluecke, B. Spatial and angular variational super-resolution of 4D light fields. In: Computer Vision – ECCV 2012. Lecture Notes in Computer Science, Vol. 7576. Fitzgibbon, A.; Lazebnik, S.; Perona, P.; Sato, Y.; Schmid, C. Eds. Springer Berlin, Heidelberg, 608–621, 2012.
Crossref
[12]
Wanner, S.; Goldluecke, B. Variational light field analysis for disparity estimation and super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 36, No. 3, 606–619, 2014.
Crossref Google Scholar
[13]
Mitra, K.; Veeraraghavan, A. Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 22–28, 2012.
Crossref
[14]
Yuan, Y.; Cao, Z. Q.; Su, L. J. Light-field image superresolution using a combined deep CNN based on EPI. IEEE Signal Processing Letters Vol. 25, No. 9, 1359–1363, 2018.
Crossref Google Scholar
[15]
Yoon, Y.; Jeon, H. G.; Yoo, D.; Lee, J. Y.; Kweon, I. S. Learning a deep convolutional network for light-field image super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision Workshop, 57–65, 2015.
Crossref
[16]
Yoon, Y.; Jeon, H. G.; Yoo, D.; Lee, J. Y.; Kweon, I. S. Light-field image super-resolution using convolutional neural network. IEEE Signal Processing Letters Vol. 24, No. 6, 848–852, 2017.
Crossref Google Scholar
[17]
Li, D. L.; Yang, D.; Wang, S. Z.; Sheng, H. Light field super-resolution based on spatial and angular attention. In: Wireless Algorithms, Systems, and Applications. Lecture Notes in Computer Science, Vol. 12937. Liu, Z.; Wu, F.; Das, S. K. Eds. Springer Cham, 314–325, 2021.
Crossref
[18]
Zhang, S.; Lin, Y. F.; Sheng, H. Residual networks for light field image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11038–11047, 2019.
Crossref
[19]
Yeung, H. W. F.; Hou, J. H.; Chen, X. M.; Chen, J.; Chen, Z. B.; Chung, Y. Y. Light field spatial super-resolution using deep efficient spatial-angular separable convolution. IEEE Transactions on Image Processing Vol. 28, No. 5, 2319–2330, 2019.
Crossref Google Scholar
[20]
Hu, J.; Shen, L.; Sun, G. Squeeze-and-excitation networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7132–7141, 2018.
Crossref
[21]
Wang, L. G.; Wang, Y. Q.; Liang, Z. F.; Lin, Z. P.; Yang, J. G.; An, W.; Guo, Y. L. Learning parallax attention for stereo image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12242–12251, 2019.
Crossref
[22]
Wang, Y. Q.; Ying, X. Y.; Wang, L. G.; Yang, J. G.; An, W.; Guo, Y. L. Symmetric parallax attention for stereo image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 766–775, 2021.
Crossref
[23]
Yang, W. M.; Zhang, X. C.; Tian, Y. P.; Wang, W.; Xue, J. H.; Liao, Q. M. Deep learning for single image super-resolution: A brief review. IEEE Transactions on Multimedia Vol. 21, No. 12, 3106–3121, 2019.
Crossref Google Scholar
[24]
Wang, Z. H.; Chen, J.; Hoi, S. C. H. Deep learning for image super-resolution: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 43, No. 10, 3365–3387, 2021.
Crossref Google Scholar
[25]
Dong, C.; Loy, C. C.; He, K. M.; Tang, X. O. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 38, No. 2, 295–307, 2015.
Crossref Google Scholar
[26]
Dong, C.; Loy, C. C.; He, K. M.; Tang, X. O. Learning a deep convolutional network for image super-resolution. In: Computer Vision – ECCV 2014. Lecture Notes in Computer Science, Vol. 8692. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer Cham, 184–199, 2014.
Crossref
[27]
Kim, J.; Lee, J. K.; Lee, K. M. Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1646–1654, 2016.
Crossref
[28]
Lim, B.; Son, S.; Kim, H.; Nah, S.; Lee, K. M. Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 1132–1140, 2017.
Crossref
[29]
Timofte, R.; Agustsson, E.; Van Gool, L.; Yang, M.-H.; Zhang, L. NTIRE 2017 challenge on single image super-resolution: Methods and results. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 1110–1121, 2017.
Crossref
[30]
Zhang, Y. L.; Tian, Y. P.; Kong, Y.; Zhong, B. N.; Fu, Y. Residual dense network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2472–2481, 2018.
Crossref
[31]
Zhang, Y. L.; Tian, Y. P.; Kong, Y.; Zhong, B. N.; Fu, Y. Residual dense network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 43, No. 7, 2480–2495, 2021.
Crossref Google Scholar
[32]
Zhang, Y. L.; Li, K. P.; Li, K.; Wang, L. C.; Zhong, B. N.; Fu, Y. Image super-resolution using very deep residual channel attention networks. In: Computer Vision – ECCV 2018. Lecture Notes in Computer Science, Vol. 11211. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 294–310, 2018.
Crossref
[33]
Dai, T.; Cai, J. R.; Zhang, Y. B.; Xia, S. T.; Zhang, L. Second-order attention network for single image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11057–11066, 2019.
Crossref
[34]
Zhang, F. L.; Wang, J.; Shechtman, E.; Zhou, Z. Y.; Shi, J. X.; Hu, S. M. PlenoPatch: Patch-based plenoptic image manipulation. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 5, 1561–1573, 2017.
Crossref Google Scholar
[35]
Rossi, M.; Frossard, P. Geometry-consistent light field super-resolution via graph-based regularization. IEEE Transactions on Image Processing Vol. 27, No. 9, 4207–4218, 2018.
Crossref Google Scholar
[36]
Alain, M.; Smolic, A. Light field super-resolution via LFBM5D sparse coding. In: Proceedings of the 25th IEEE International Conference on Image Processing, 2501–2505, 2018.
Crossref
[37]
Huang, Y.; Wang, W.; Wang, L. Bidirectional recurrent convolutional networks for multi-frame super-resolution. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, 235–243, 2015.
[38]
Wang, Y. L.; Liu, F.; Zhang, K. B.; Hou, G. Q.; Sun, Z. N.; Tan, T. N. LFNet: A novel bidirectional recurrent convolutional neural network for light-field image super-resolution. IEEE Transactions on Image Processing Vol. 27, No. 9, 4274–4286, 2018.
Crossref Google Scholar
[39]
Jin, J.; Hou, J. H.; Chen, J.; Kwong, S. Light field spatial super-resolution via deep combinatorialgeometry embedding and structural consistency regularization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2257–2266, 2020.
Crossref
[40]
Wang, Y. Q.; Wang, L. G.; Yang, J. G.; An, W.; Yu, J. Y.; Guo, Y. L. Spatial-angular interaction for light field image super-resolution. In: Computer Vision – ECCV 2020. Lecture Notes in Computer Science, Vol. 12368. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 290–308, 2020.
Crossref
[41]
Mo, Y.; Wang, Y.; Xiao, C.; Yang, J.; An, W. Dense dual-attention network for light field image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology Vol. 32, No. 7, 4431–4443, 2022.
Crossref Google Scholar
[42]
Rerabek, M.; Ebrahimi, T. New light field image dataset. In: Proceedings of the 8th International Conference on Quality of Multimedia Experience, 2016.
[43]
Honauer, K.; Johannsen, O.; Kondermann, D.; Goldluecke, B. A dataset and evaluation methodology for depth estimation on 4D light fields. In: Computer Vision – ACCV 2016. Lecture Notes in Computer Science, Vol. 10113. Lai, S. H.; Lepetit, V.; Nishino, K.; Sato, Y. Eds. Springer Cham, 19–34, 2017.
Crossref
[44]
Wanner, S.; Meister, S.; Goldluecke, B. Datasets and benchmarks for densely sampled 4D light fields. In: Proceedings of the Vision, Modeling & Visualization, 225–226, 2013.
[45]
Le Pendu, M.; Jiang, X. R.; Guillemot, C. Light field inpainting propagation via low rank matrix completion. IEEE Transactions on Image Processing Vol. 27, No. 4, 1981–1993, 2018.
Crossref Google Scholar
[46]
Vaish, V.; Adams, A. The (new) Stanford light field archive. Computer Graphics Laboratory, Stanford University, 2008. Available at http://lightfield.stanford.edu/index.html.
[47]
Raj, A. S.; Lowney, M.; Shah, R.; Wetzstein, G. Stanford Lytro light field archive. 2016. Available at http://lightfields.stanford.edu/LF2016.html.
[48]
Chen, L. C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A. L. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 40, No. 4, 834–848, 2018.
Crossref Google Scholar
[49]
Ying, X. Y.; Wang, Y. Q.; Wang, L. G.; Sheng, W. D.; An, W.; Guo, Y. L. A stereo attention module for stereo image super-resolution. IEEE Signal Processing Letters Vol. 27, 496–500, 2020.
Crossref Google Scholar
[50]
Kim, J.; Lee, J. K.; Lee, K. M. Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1646–1654, 2016.
Crossref
Computational Visual Media
Volume 9 Issue 4,
December 2023
Pages 843-858
DOI: 10.1007/s41095-022-0297-1
Cite this article:
Zhang W, Ke W, Yang D, et al. Light field super-resolution using complementary-view feature attention. Computational Visual Media, 2023, 9(4): 843-858. https://doi.org/10.1007/s41095-022-0297-1

618

Views

15

Downloads

8

Crossref

9

Web of Science

11

Scopus

0

CSCD

Altmetrics
Received: 24 February 2022
Accepted: 11 June 2022
Published: 03 July 2023
© The Author(s) 2023.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
Return