Home Computational Visual Media Article
PDF (12.4 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript
Show Outline
Figures (11)

Show 2 more figures Hide 2 figures
Tables (4)
Table 1
Table 2
Table 3
Table 4
Research Article | Open Access

RecStitchNet: Learning to stitch images with rectangular boundaries

Key Lab of Film and TV Media Technology of Zhejiang Province, College of Media Engineering, Communication University of Zhejiang, Hangzhou 310018, China
School of Computer Science and Informatics, Cardiff University, Cardiff CF24 4AG, UK
Institute of Information Science, Beijing Jiaotong University, Beijing 100091, China
School of Engineering and Computer Science, Victoria University of Wellington, Wellington 6012, New Zealand
STEM, University of South Australia, Adelaide 5095, Australia
Show Author Information

Graphical Abstract

View original image Download original image

Abstract

Irregular boundaries in image stitching naturally occur due to freely moving cameras. To deal with this problem, existing methods focus on optimizing mesh warping to make boundaries regular using the traditional explicit solution. However, previous methods always depend on hand-crafted features (e.g., keypoints and line segments). Thus, failures often happen in overlapping regions without distinctive features. In this paper, we address this problem by proposing RecStitchNet, a reasonable and effective network for image stitching with rectangular boundaries. Considering that both stitching and imposing rectangularity are non-trivial tasks in the learning-based framework, we propose a three-step progressive learning based strategy, which not only simplifies this task, but gradually achieves a good balance between stitching and imposing rectangularity. In the first step, we perform initial stitching by a pre-trained state-of-the-art image stitching model, to produce initially warped stitching results without considering the boundary constraint. Then, we use a regression network with a comprehensive objective regarding mesh, perception, and shape to further encourage the stitched meshes to have rectangular boundaries with high content fidelity. Finally, we propose an unsupervised instance-wise optimization strategy to refine the stitched meshes iteratively, which can effectively improve the stitching results in terms of feature alignment, as well as boundary and structure preservation. Due to the lack of stitching datasets and the difficulty of label generation, we propose to generate a stitching dataset with rectangular stitched images as pseudo-ground-truth labels, and the performance upper bound induced from the it can be broken by our unsupervised refinement. Qualitative and quantitative results and evaluations demonstrate the advantages of our method over the state-of-the-art.

Keywords

image stitchingboundariesconvolutional neural network

References

[1]
Chen, Y. S.; Chuang, Y. Y. Natural image stitching with the global similarity prior. In: Computer Vision – ECCV 2016. Lecture Notes in Computer Science, Vol. 9909. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 186–201, 2016.
Crossref
[2]
Gao, J.; Kim, S. J.; Brown, M. S. Constructing image panoramas using dual-homography warping. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, 49–56, 2011.
Crossref
[3]
Lin, C. C.; Pankanti, S. U.; Ramamurthy, K. N.; Aravkin, A. Y. Adaptive as-natural-as-possible image stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1155–1163, 2015.
Crossref
[4]
Nie, L.; Lin, C.; Liao, K.; Liu, M.; Zhao, Y. A view-free image stitching network based on global homography. Journal of Visual Communication and Image Representation Vol. 73, Article No. 102950, 2020.
Crossref
[5]

Zhao, Q.; Ma, Y.; Zhu, C.; Yao, C.; Feng, B.; Dai, F. Image stitching via deep homography estimation. Neurocomputing Vol. 450, 219–229, 2021.

Crossref Google Scholar
[6]

Kweon, H.; Kim, H.; Kang, Y.; Yoon, Y.; Jeong, W.; Yoon, K. J. Pixel-wise warping for deep image stitching. Proceedings of the AAAI Conference on Artificial Intelligence Vol. 37, No. 1, 1196–1204, 2023.

Crossref Google Scholar
[7]
Song, D. Y.; Lee, G.; Lee, H.; Um, G. M.; Cho, D. Weakly-supervised stitching network for real-world panoramic image generation. In: Computer Vision – ECCV 2022. Lecture Notes in Computer Science, Vol. 13676. Avidan, S.; Brostow, G.; Cissé, M.; Farinella, G. M.; Hassner, T. Eds. Springer Cham, 54–71, 2022.
Crossref
[8]

Nie, L.; Lin, C.; Liao, K.; Liu, S.; Zhao, Y. Unsupervised deep image stitching: Reconstructing stitched features to images. IEEE Transactions on Image Processing Vol. 30, 6184–6197, 2021.

Crossref Google Scholar
[9]
Nie, L.; Lin, C.; Liao, K.; Liu, S.; Zhao, Y. Parallax-tolerant unsupervised deep image stitching. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 7365–7374, 2023.
Crossref
[10]
He, K.; Chang, H.; Sun, J. Rectangling panoramic images via warping. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 79, 2013.
Crossref
[11]

Zhang, Y.; Lai, Y. K.; Zhang, F. L. Content-preserving image stitching with piecewise rectangular boundary constraints. IEEE Transactions on Visualization and Computer Graphics Vol. 27, No. 7, 3198–3212, 2021.

Crossref Google Scholar
[12]
Nie, L.; Lin, C.; Liao, K.; Liu, S.; Zhao, Y. Deep rectangling for image stitching: A learning baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5730–5738, 2022.
Crossref
[13]

Nie, L.; Lin, C.; Liao, K.; Liu, S.; Zhao, Y. Deep rotation correction without angle prior. IEEE Transactions on Image Processing Vol. 32, 2879–2888, 2023.

Crossref Google Scholar
[14]

Brown, M.; Lowe, D. G. Automatic panoramic image stitching using invariant features. International Journal of Computer Vision Vol. 74, No. 1, 59–73, 2007.

Crossref Google Scholar
[15]
Lin, W. Y.; Liu, S.; Matsushita, Y.; Ng, T. T.; Cheong, L. F. Smoothly varying affine stitching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 345–352, 2011.
Crossref
[16]

Zaragoza, J.; Chin, T. J.; Tran, Q. H.; Brown, M. S.; Suter, D. As-projective-as-possible image stitching with moving DLT. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 36, No. 7, 1285–1298, 2014.

Crossref Google Scholar
[17]

Lou, Z.; Gevers, T. Image alignment by piecewise planar region matching. IEEE Transactions on Multimedia Vol. 16, No. 7, 2052–2061, 2014.

Crossref Google Scholar
[18]
Lin, K.; Jiang, N.; Cheong, L. F.; Do, M.; Lu, J. SEAGULL: seam-guided local alignment for parallax-tolerant image stitching. In: Computer Vision – ECCV 2016. Lecture Notes in Computer Science, Vol. 9907. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 370–385, 2016.
Crossref
[19]
Zhang, F.; Liu, F. Parallax-tolerant image stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3262–3269, 2014.
Crossref
[20]
Gao, J.; Li, Y.; Chin, T.; Brown, M. S. Seam-driven image stitching. In: Proceedings of the 34th Annual Conference of the European Association for Computer Graphics, 2013.
[21]
Chang, C. H.; Sato, Y.; Chuang, Y. Y. Shape-preserving half-projective warps for image stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3254–3261, 2014.
Crossref
[22]

Li, N.; Xu, Y.; Wang, C. Quasi-homography warps in image stitching. IEEE Transactions on Multimedia Vol. 20, No. 6, 1365–1375, 2018.

Crossref Google Scholar
[23]

Liao, T.; Li, N. Single-perspective warps in natural image stitching. IEEE Transactions on Image Processing Vol. 29, 724–735, 2020.

Crossref Google Scholar
[24]
Du, P.; Ning, J.; Cui, J.; Huang, S.; Wang, X.; Wang, J. Geometric structure preserving warp for natural image stitching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3678–3686, 2022.
Crossref
[25]
Jia, Q.; Li, Z.; Fan, X.; Zhao, H.; Teng, S.; Ye, X.; Latecki, L. J. Leveraging line-point consistence to preserve structures for wide parallax image stitching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12181–12190, 2021.
Crossref
[26]

Zhang, L.; Huang, H. Image stitching with manifold optimization. IEEE Transactions on Multimedia Vol. 25, 3469–3482, 2023.

Crossref Google Scholar
[27]

Li, N.; Liao, T.; Wang, C. Perception-based seam cutting for image stitching. Signal, Image and Video Processing Vol. 12, No. 5, 967–974, 2018.

Crossref Google Scholar
[28]
Lai, W. S.; Gallo, O.; Gu, J.; Sun, D.; Yang, M. H.; Kautz, J. Video stitching for linear camera arrays. In: Proceedings of the 30th British Machine Vision Conference, 2019.
[29]
He, K.; Chang, H.; Sun, J. Content-aware rotation. In: Proceedings of the IEEE International Conference on Computer Vision, 553–560, 2013.
Crossref
[30]

Wu, J. L.; Shi, J. J.; Zhang, L. Rectangling irregular videos by optimal spatio-temporal warping. Computational Visual Media Vol. 8, No. 1, 93–103, 2022.

Crossref Google Scholar
[31]
Liao, K.; Nie, L.; Lin, C.; Zheng, Z.; Zhao, Y. RecRecNet: Rectangling rectified wide-angle images by thin-plate spline model and DoF-based curriculum learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 10766–10775, 2023.
Crossref
[32]
Zhou, H.; Zhu, Y.; Lv, X.; Liu, Q.; Zhang, S. Rectangular-output image stitching. In: Proceedings of the IEEE International Conference on Image Processing, 2800–2804, 2023.
Crossref
[33]
Jaderberg, M.; Simonyan, K.; Zisserman, A.; Kavukcuoglu, K. Spatial transformer networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, Vol. 2, 2017–2025, 2015.
[34]
Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. In: Proceedings of the 3rd International Conference on Learning Representations, 2015.
[35]

Martínez, F.; Rueda, A. J.; Feito, F. R. A new algorithm for computing Boolean operations on polygons. Computers & Geosciences Vol. 35, No. 6, 1177–1185, 2009.

Crossref Google Scholar
[36]

Wang, M.; Shamir, A.; Yang, G. Y.; Lin, J. K.; Yang, G. W.; Lu, S. P.; Hu, S. M. BiggerSelfie: Selfie video expansion with hand-held camera. IEEE Transactions on Image Processing Vol. 27, No. 12, 5854–5865, 2018.

Crossref Google Scholar
[37]

Nie, Y.; Su, T.; Zhang, Z.; Sun, H.; Li, G. Dynamic video stitching via shakiness removing. IEEE Transactions on Image Processing Vol. 27, No. 1, 164–178, 2018.

Crossref Google Scholar
[38]

Wang, M.; Yang, G. Y.; Lin, J. K.; Zhang, S. H.; Shamir, A.; Lu, S. P.; Hu, S. M. Deep online video stabilization with multi-grid warping transformation learning. IEEE Transactions on Image Processing Vol. 28, No. 5, 2283–2292, 2019.

Crossref Google Scholar
[39]

Rong, J. X.; Zhang, L.; Huang, H.; Zhang, F. L. IMU-assisted online video background identification. IEEE Transactions on Image Processing Vol. 31, 4336–4351, 2022.

Crossref Google Scholar
Computational Visual Media
Volume 10 Issue 4,
August 2024
Pages 687-703
DOI: 10.1007/s41095-024-0420-6
Cite this article:
Zhang Y, Lai Y-K, Nie L, et al. RecStitchNet: Learning to stitch images with rectangular boundaries. Computational Visual Media, 2024, 10(4): 687-703. https://doi.org/10.1007/s41095-024-0420-6
Metrics & Citations  
Article History
Copyright
Rights and Permissions
Return

Metrics for examples in Figs. 7 and 8

Metrics Fig. 7(1) Fig. 7(2) Fig. 8(1) Fig. 8(2) Fig. 8(3) Fig. 8(4) Fig. 8(5) Fig. 8(6)
Nie et al. (2022)
PSNR 25.7282 26.8942 28.3332 27.8733 28.3954 30.1785 30.6975 29.9269
SSIM 0.9442 0.9390 0.9522 0.9430 0.9278 0.9490 0.9668 0.9499
Mask 0.9905 0.9873 0.9895 0.9873 0.9906 0.9925 0.9944 0.9927
Zhang et al. (2021)
PSNR 23.3887 23.1461 25.9588 24.9477 26.6624 27.2616 27.8547 28.2692
SSIM 0.8947 0.8597 0.9161 0.8844 0.9102 0.8901 0.9335 0.9171
Mask 0.9903 0.9908 0.9920 0.9924 0.9921 0.9920 0.9919 0.9920
Ours
PSNR 25.4717 26.0382 29.0105 30.0713 28.8899 33.6945 32.9202 32.0031
SSIM 0.9618 0.9098 0.9619 0.9394 0.9454 0.9647 0.9660 0.9486
Mask 0.9951 0.9935 0.9936 0.9950 0.9940 0.9932 0.9947 0.9934

Quantitative evaluation on the testing dataset of UDIS-D. The upper part and the lower part give quantitative evaluations on the selected 1068 examples and the remaining 38 examples of the testing dataset of UDIS-D

Metrics Pseudo-ground-truth Nie et al. (2022) Ours Ours+refinement
Selected 1068 testing examples of UDIS-D
PSNR 25.0656 25.9212 21.3544 27.7812
SSIM 0.8454 0.8581 0.7020 0.8958
Mask 0.9903 0.9913 0.9889 0.9941
Boundary 0.0002 0.00017 0.0016 0.00014
Remaining 38 testing examples of UDIS-D
PSNR — 24.7549 20.9781 26.7898
SSIM — 0.8292 0.7281 0.8772
Mask — 0.9909 0.9742 0.9920

Ablation study metrics

Metric w/o shape w/o perception w/o mesh w/o correlation Our method
PSNR 18.1769 21.0704 17.8817 20.7431 21.3544
SSIM 0.6147 0.6690 0.5753 0.6758 0.7020
Mask 0.9617 0.9828 0.9937 0.9814 0.9889
Boundary 0.0382 0.0017 0.0756 0.0020 0.0016

Average running time (Unit: s)

Nie et al. Zhang et al. Ours Ours+refinement
0.256 1.413 0.211 1.921