AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Computational Visual Media Article
PDF (12.4 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Research Article | Open Access

RecStitchNet: Learning to stitch images with rectangular boundaries

Key Lab of Film and TV Media Technology of Zhejiang Province, College of Media Engineering, Communication University of Zhejiang, Hangzhou 310018, China
School of Computer Science and Informatics, Cardiff University, Cardiff CF24 4AG, UK
Institute of Information Science, Beijing Jiaotong University, Beijing 100091, China
School of Engineering and Computer Science, Victoria University of Wellington, Wellington 6012, New Zealand
STEM, University of South Australia, Adelaide 5095, Australia
Show Author Information

Graphical Abstract

Abstract

Irregular boundaries in image stitching naturally occur due to freely moving cameras. To deal with this problem, existing methods focus on optimizing mesh warping to make boundaries regular using the traditional explicit solution. However, previous methods always depend on hand-crafted features (e.g., keypoints and line segments). Thus, failures often happen in overlapping regions without distinctive features. In this paper, we address this problem by proposing RecStitchNet, a reasonable and effective network for image stitching with rectangular boundaries. Considering that both stitching and imposing rectangularity are non-trivial tasks in the learning-based framework, we propose a three-step progressive learning based strategy, which not only simplifies this task, but gradually achieves a good balance between stitching and imposing rectangularity. In the first step, we perform initial stitching by a pre-trained state-of-the-art image stitching model, to produce initially warped stitching results without considering the boundary constraint. Then, we use a regression network with a comprehensive objective regarding mesh, perception, and shape to further encourage the stitched meshes to have rectangular boundaries with high content fidelity. Finally, we propose an unsupervised instance-wise optimization strategy to refine the stitched meshes iteratively, which can effectively improve the stitching results in terms of feature alignment, as well as boundary and structure preservation. Due to the lack of stitching datasets and the difficulty of label generation, we propose to generate a stitching dataset with rectangular stitched images as pseudo-ground-truth labels, and the performance upper bound induced from the it can be broken by our unsupervised refinement. Qualitative and quantitative results and evaluations demonstrate the advantages of our method over the state-of-the-art.

Keywords

image stitchingboundariesconvolutional neural network

References

[1]
Chen, Y. S.; Chuang, Y. Y. Natural image stitching with the global similarity prior. In: Computer Vision – ECCV 2016. Lecture Notes in Computer Science, Vol. 9909. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 186–201, 2016.
Crossref
[2]
Gao, J.; Kim, S. J.; Brown, M. S. Constructing image panoramas using dual-homography warping. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, 49–56, 2011.
Crossref
[3]
Lin, C. C.; Pankanti, S. U.; Ramamurthy, K. N.; Aravkin, A. Y. Adaptive as-natural-as-possible image stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1155–1163, 2015.
Crossref
[4]
Nie, L.; Lin, C.; Liao, K.; Liu, M.; Zhao, Y. A view-free image stitching network based on global homography. Journal of Visual Communication and Image Representation Vol. 73, Article No. 102950, 2020.
Crossref
[5]

Zhao, Q.; Ma, Y.; Zhu, C.; Yao, C.; Feng, B.; Dai, F. Image stitching via deep homography estimation. Neurocomputing Vol. 450, 219–229, 2021.
Crossref Google Scholar
[6]

Kweon, H.; Kim, H.; Kang, Y.; Yoon, Y.; Jeong, W.; Yoon, K. J. Pixel-wise warping for deep image stitching. Proceedings of the AAAI Conference on Artificial Intelligence Vol. 37, No. 1, 1196–1204, 2023.
Crossref Google Scholar
[7]
Song, D. Y.; Lee, G.; Lee, H.; Um, G. M.; Cho, D. Weakly-supervised stitching network for real-world panoramic image generation. In: Computer Vision – ECCV 2022. Lecture Notes in Computer Science, Vol. 13676. Avidan, S.; Brostow, G.; Cissé, M.; Farinella, G. M.; Hassner, T. Eds. Springer Cham, 54–71, 2022.
Crossref
[8]

Nie, L.; Lin, C.; Liao, K.; Liu, S.; Zhao, Y. Unsupervised deep image stitching: Reconstructing stitched features to images. IEEE Transactions on Image Processing Vol. 30, 6184–6197, 2021.
Crossref Google Scholar
[9]
Nie, L.; Lin, C.; Liao, K.; Liu, S.; Zhao, Y. Parallax-tolerant unsupervised deep image stitching. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 7365–7374, 2023.
Crossref
[10]
He, K.; Chang, H.; Sun, J. Rectangling panoramic images via warping. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 79, 2013.
Crossref
[11]

Zhang, Y.; Lai, Y. K.; Zhang, F. L. Content-preserving image stitching with piecewise rectangular boundary constraints. IEEE Transactions on Visualization and Computer Graphics Vol. 27, No. 7, 3198–3212, 2021.
Crossref Google Scholar
[12]
Nie, L.; Lin, C.; Liao, K.; Liu, S.; Zhao, Y. Deep rectangling for image stitching: A learning baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5730–5738, 2022.
Crossref
[13]

Nie, L.; Lin, C.; Liao, K.; Liu, S.; Zhao, Y. Deep rotation correction without angle prior. IEEE Transactions on Image Processing Vol. 32, 2879–2888, 2023.
Crossref Google Scholar
[14]

Brown, M.; Lowe, D. G. Automatic panoramic image stitching using invariant features. International Journal of Computer Vision Vol. 74, No. 1, 59–73, 2007.
Crossref Google Scholar
[15]
Lin, W. Y.; Liu, S.; Matsushita, Y.; Ng, T. T.; Cheong, L. F. Smoothly varying affine stitching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 345–352, 2011.
Crossref
[16]

Zaragoza, J.; Chin, T. J.; Tran, Q. H.; Brown, M. S.; Suter, D. As-projective-as-possible image stitching with moving DLT. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 36, No. 7, 1285–1298, 2014.
Crossref Google Scholar
[17]

Lou, Z.; Gevers, T. Image alignment by piecewise planar region matching. IEEE Transactions on Multimedia Vol. 16, No. 7, 2052–2061, 2014.
Crossref Google Scholar
[18]
Lin, K.; Jiang, N.; Cheong, L. F.; Do, M.; Lu, J. SEAGULL: seam-guided local alignment for parallax-tolerant image stitching. In: Computer Vision – ECCV 2016. Lecture Notes in Computer Science, Vol. 9907. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 370–385, 2016.
Crossref
[19]
Zhang, F.; Liu, F. Parallax-tolerant image stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3262–3269, 2014.
Crossref
[20]
Gao, J.; Li, Y.; Chin, T.; Brown, M. S. Seam-driven image stitching. In: Proceedings of the 34th Annual Conference of the European Association for Computer Graphics, 2013.
[21]
Chang, C. H.; Sato, Y.; Chuang, Y. Y. Shape-preserving half-projective warps for image stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3254–3261, 2014.
Crossref
[22]

Li, N.; Xu, Y.; Wang, C. Quasi-homography warps in image stitching. IEEE Transactions on Multimedia Vol. 20, No. 6, 1365–1375, 2018.
Crossref Google Scholar
[23]

Liao, T.; Li, N. Single-perspective warps in natural image stitching. IEEE Transactions on Image Processing Vol. 29, 724–735, 2020.
Crossref Google Scholar
[24]
Du, P.; Ning, J.; Cui, J.; Huang, S.; Wang, X.; Wang, J. Geometric structure preserving warp for natural image stitching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3678–3686, 2022.
Crossref
[25]
Jia, Q.; Li, Z.; Fan, X.; Zhao, H.; Teng, S.; Ye, X.; Latecki, L. J. Leveraging line-point consistence to preserve structures for wide parallax image stitching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12181–12190, 2021.
Crossref
[26]

Zhang, L.; Huang, H. Image stitching with manifold optimization. IEEE Transactions on Multimedia Vol. 25, 3469–3482, 2023.
Crossref Google Scholar
[27]

Li, N.; Liao, T.; Wang, C. Perception-based seam cutting for image stitching. Signal, Image and Video Processing Vol. 12, No. 5, 967–974, 2018.
Crossref Google Scholar
[28]
Lai, W. S.; Gallo, O.; Gu, J.; Sun, D.; Yang, M. H.; Kautz, J. Video stitching for linear camera arrays. In: Proceedings of the 30th British Machine Vision Conference, 2019.
[29]
He, K.; Chang, H.; Sun, J. Content-aware rotation. In: Proceedings of the IEEE International Conference on Computer Vision, 553–560, 2013.
Crossref
[30]

Wu, J. L.; Shi, J. J.; Zhang, L. Rectangling irregular videos by optimal spatio-temporal warping. Computational Visual Media Vol. 8, No. 1, 93–103, 2022.
Crossref Google Scholar
[31]
Liao, K.; Nie, L.; Lin, C.; Zheng, Z.; Zhao, Y. RecRecNet: Rectangling rectified wide-angle images by thin-plate spline model and DoF-based curriculum learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 10766–10775, 2023.
Crossref
[32]
Zhou, H.; Zhu, Y.; Lv, X.; Liu, Q.; Zhang, S. Rectangular-output image stitching. In: Proceedings of the IEEE International Conference on Image Processing, 2800–2804, 2023.
Crossref
[33]
Jaderberg, M.; Simonyan, K.; Zisserman, A.; Kavukcuoglu, K. Spatial transformer networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, Vol. 2, 2017–2025, 2015.
[34]
Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. In: Proceedings of the 3rd International Conference on Learning Representations, 2015.
[35]

Martínez, F.; Rueda, A. J.; Feito, F. R. A new algorithm for computing Boolean operations on polygons. Computers & Geosciences Vol. 35, No. 6, 1177–1185, 2009.
Crossref Google Scholar
[36]

Wang, M.; Shamir, A.; Yang, G. Y.; Lin, J. K.; Yang, G. W.; Lu, S. P.; Hu, S. M. BiggerSelfie: Selfie video expansion with hand-held camera. IEEE Transactions on Image Processing Vol. 27, No. 12, 5854–5865, 2018.
Crossref Google Scholar
[37]

Nie, Y.; Su, T.; Zhang, Z.; Sun, H.; Li, G. Dynamic video stitching via shakiness removing. IEEE Transactions on Image Processing Vol. 27, No. 1, 164–178, 2018.
Crossref Google Scholar
[38]

Wang, M.; Yang, G. Y.; Lin, J. K.; Zhang, S. H.; Shamir, A.; Lu, S. P.; Hu, S. M. Deep online video stabilization with multi-grid warping transformation learning. IEEE Transactions on Image Processing Vol. 28, No. 5, 2283–2292, 2019.
Crossref Google Scholar
[39]

Rong, J. X.; Zhang, L.; Huang, H.; Zhang, F. L. IMU-assisted online video background identification. IEEE Transactions on Image Processing Vol. 31, 4336–4351, 2022.
Crossref Google Scholar
Computational Visual Media
Volume 10 Issue 4,
August 2024
Pages 687-703
DOI: 10.1007/s41095-024-0420-6
Cite this article:
Zhang Y, Lai Y-K, Nie L, et al. RecStitchNet: Learning to stitch images with rectangular boundaries. Computational Visual Media, 2024, 10(4): 687-703. https://doi.org/10.1007/s41095-024-0420-6

105

Views

4

Downloads

1

Crossref

1

Web of Science

1

Scopus

0

CSCD

Altmetrics
Received: 05 January 2024
Accepted: 27 February 2024
Published: 06 August 2024
© The Author(s) 2024.

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
Return