AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Computational Visual Media Article
PDF (7.5 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Research Article | Open Access

Multi-task learning and joint refinement between camera localization and object detection

Junyi Wang1,2, Yue Qi1,2,3( )
State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University, Beijing 100191, China
Peng Cheng Laboratory, Shenzhen 518052, China
Qingdao Research Institute of Beihang University, Qingdao 266104, China

* Junyi Wang’s present address: School of Computer Science and Technology, Shandong University, Qingdao, China.

Show Author Information

Graphical Abstract

Abstract

Visual localization and object detection both play important roles in various tasks. In many indoor application scenarios where some detected objects have fixed positions, the two techniques work closely together. However, few researchers consider these two tasks simultaneously, because of a lack of datasets and the little attention paid to such environments. In this paper, we explore multi-task network design and joint refinement of detection and localization. To address the dataset problem, we construct a medium indoor scene of an aviation exhibition hall through a semi-automatic process. The dataset provides localization and detection information, and is publicly available at https://drive.google.com/drive/folders/1U28zkON4_I0dbzkqyIAKlAl5k9oUK0jI?usp=sharing for benchmarking localization and object detection tasks. Targeting this dataset, we have designed a multi-task network, JLDNet, based on YOLO v3, that outputs a target point cloud and object bounding boxes. For dynamic environments, the detection branch also promotes the perception of dynamics. JLDNet includes image feature learning, point feature learning, feature fusion, detection construction, and point cloud regression. Moreover, object-level bundle adjustment is used to further improve localization and detection accuracy. To test JLDNet and compare it to other methods, we have conducted experiments on 7 static scenes, our constructed dataset, and the dynamic TUM RGB-D and Bonn datasets. Our results show state-of-the-art accuracy for both tasks, and the benefit of jointly working on both tasks is demonstrated.

Keywords

visual localizationobject detectionjoint optimizationmulti-task learning

References

[1]

Bao, W.; Wang, W.; Xu, Y. H.; Guo, Y. L.; Hong, S. Y.; Zhang, X. H. InStereo2K: A large real dataset for stereo matching in indoor scenes. Science China Information Sciences Vol. 63, No. 11, 212101, 2020.
Crossref Google Scholar
[2]

Yan, F. H.; Li, Z. X.; Zhou, Z. Robust and efficient edge-based visual odometry. Computational Visual Media Vol. 8, No. 3, 467–481, 2022.
Crossref Google Scholar
[3]

Huang, J. H.; Yang, S.; Zhao, Z. S.; Lai, Y. K.; Hu, S. M. ClusterSLAM: A SLAM backend for simultaneous rigid body clustering and motion estimation. Computational Visual Media Vol. 7, No. 1, 87–101, 2021.
Crossref Google Scholar
[4]

Wang, C.; Guo, X. H. Feature-based RGB-D camera pose optimization for real-time 3D reconstruction. Computational Visual Media Vol. 3, No. 2, 95–106, 2017.
Crossref Google Scholar
[5]

Nakajima, Y.; Saito, H. Robust camera pose estimation by viewpoint classification using deep learning. Computational Visual Media Vol. 3, No. 2, 189–198, 2017.
Crossref Google Scholar
[6]

Liu, S.; Zhang, Y. Q.; Yang, X. S.; Shi, D. M.; Zhang, J. J. Robust facial landmark detection and tracking across poses and expressions for in-the-wild monocular video. Computational Visual Media Vol. 3, No. 1, 33–47, 2017.
Crossref Google Scholar
[7]

Qin, T.; Li, P. L.; Shen, S. J. VINS-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics Vol. 34, No. 4, 1004–1020, 2018.
Crossref Google Scholar
[8]
Rublee, E.; Rabaud, V.; Konolige, K.; Bradski, G. ORB: An efficient alternative to SIFT or SURF. In: Proceedings of the International Conference on Computer Vision, 2564–2571, 2011.
Crossref
[9]

Campos, C.; Elvira, R.; Rodriguez, J. J. G.; Montiel, J. M. M.; Tardos, J. D. ORB-SLAM3: An accurate open-source library for visual, visual–inertial, and multimap SLAM. IEEE Transactions on Robotics Vol. 37, No. 6, 1874–1890, 2021.
Crossref Google Scholar
[10]
Xu, B. B.; Li, W. B.; Tzoumanikas, D.; Bloesch, M.; Davison, A.; Leutenegger, S. MID-fusion: Octree-based object-level multi-instance dynamic SLAM. In: Proceedings of the International Conference on Robotics and Automation, 5231–5237, 2019.
Crossref
[11]

Yang, S. C.; Scherer, S. CubeSLAM: Monocular 3-D object SLAM. IEEE Transactions on Robotics Vol. 35, No. 4, 925–938, 2019.
Crossref Google Scholar
[12]

Bescos, B.; Facil, J. M.; Civera, J.; Neira, J. DynaSLAM: Tracking, mapping, and inpainting in dynamic scenes. IEEE Robotics and Automation Letters Vol. 3, No. 4, 4076–4083, 2018.
Crossref Google Scholar
[13]
Kendall, A.; Grimes, M.; Cipolla, R. PoseNet: A convolutional network for real-time 6-DOF camera relocalization. In: Proceedings of the IEEE International Conference on Computer Vision, 2938–2946, 2015.
Crossref
[14]
Brahmbhatt, S.; Gu, J. W.; Kim, K.; Hays, J.; Kautz, J. Geometry-aware learning of maps for camera localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2616–2625, 2018.
Crossref
[15]
Kendall, A.; Cipolla, R. Geometric loss functions for camera pose regression with deep learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6555–6564, 2017.
Crossref
[16]
Walch, F.; Hazirbas, C.; Leal-Taixe, L.; Sattler, T.; Hilsenbeck, S.; Cremers, D. Image-based localization using LSTMs for structured feature correlation. In: Proceedings of the IEEE International Conference on Computer Vision, 627–637, 2017.
Crossref
[17]
Xue, F.; Wang, X.; Yan, Z. K.; Wang, Q. Y.; Wang, J. Q.; Zha, H. B. Local supports global: Deep camera relocalization with sequence enhancement. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2841–2850, 2019.
Crossref
[18]
Kendall, A.; Cipolla, R. Modelling uncertainty in deep learning for camera relocalization. In: Proceedings of the IEEE International Conference on Robotics and Automation, 4762–4769, 2016.
Crossref
[19]

Radwan, N.; Valada, A.; Burgard, W. VLocNet++: Deep multitask learning for semantic visual localization and odometry. IEEE Robotics and Automation Letters Vol. 3, No. 4, 4407–4414, 2018.
Crossref Google Scholar
[20]
Cavallari, T.; Golodetz, S.; Lord, N. A.; Valentin, J.; Di Stefano, L.; Torr, P. H. S. On-the-fly adaptation of regression forests for online camera relocalisation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 218–227, 2017.
Crossref
[21]

Schmidt, T.; Newcombe, R.; Fox, D. Self-supervised visual descriptor learning for dense correspondence. IEEE Robotics and Automation Letters Vol. 2, No. 2, 420–427, 2017.
Crossref Google Scholar
[22]
Brachmann, E.; Michel, F.; Krull, A.; Yang, M. Y.; Gumhold, S.; Rother, C. Uncertainty-driven 6D pose estimation of objects and scenes from a single RGB image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3364–3372, 2016.
Crossref
[23]
Guzman-Rivera, A.; Kohli, P.; Glocker, B.; Shotton, J.; Sharp, T.; Fitzgibbon, A.; Izadi, S. Multi-output learning for camera relocalization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1114–1121, 2014.
Crossref
[24]
Shotton, J.; Glocker, B.; Zach, C.; Izadi, S.; Criminisi, A.; Fitzgibbon, A. Scene coordinate regression forests for camera relocalization in RGB-D images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2930–2937, 2013.
Crossref
[25]
Valentin, J.; Niebner, M.; Shotton, J.; Fitzgibbon, A.; Izadi, S.; Torr, P. Exploiting uncertainty in regression forests for accurate camera relocalization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4400–4408, 2015.
Crossref
[26]
Dong, S. Y.; Fan, Q. N.; Wang, H.; Shi, J.; Yi, L.; Funkhouser, T.; Chen, B. Q.; Guibas, L. Robust neural routing through space partitions for camera relocalization in dynamic indoor environments. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8540–8550, 2021.
Crossref
[27]

Brachmann, E.; Rother, C. Visual camera re-localization from RGB and RGB-D images using DSAC. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 44, No. 9, 5847–5865, 2022.
Google Scholar
[28]
Zhou, L.; Luo, Z. X.; Shen, T. W.; Zhang, J. H.; Zhen, M. M.; Yao, Y.; Fang, T.; Quan, L. KFNet: Learning temporal camera relocalization using Kalman filtering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4918–4927, 2020.
Crossref
[29]

Cavallari, T.; Golodetz, S.; Lord, N. A.; Valentin, J.; Prisacariu, V. A.; Stefano, L. D.; Torr, P. H. S. Real-time RGB-D camera pose estimation in novel scenes using a relocalisation cascade. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 42, No. 10, 2465–2477, 2020.
Crossref Google Scholar
[30]
Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You only look once: Unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 779–788, 2016.
Crossref
[31]
Redmon, J.; Farhadi, A. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.
[32]

Kim, S.; Park, S.; Na, B.; Yoon, S. Spiking-YOLO: Spiking neural network for energy-efficient object detection. Proceedings of the AAAI Conference on Artificial Intelligence Vol. 34, No. 7, 11270–11277, 2020.
Crossref Google Scholar
[33]
Bochkovskiy, A.; Wang, C. Y.; Liao, H. Y. M. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934, 2020.
[34]
Redmon, J.; Farhadi, A. YOLO9000: Better, faster, stronger. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6517–6525, 2017.
Crossref
[35]
Sturm, J.; Engelhard, N.; Endres, F.; Burgard, W.; Cremers, D. A benchmark for the evaluation of RGB-D SLAM systems. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 573–580, 2012.
Crossref
[36]
Palazzolo, E.; Behley, J.; Lottes, P.; Giguère, P.; Stachniss, C. ReFusion: 3D reconstruction in dynamic environments for RGB-D cameras exploiting residuals. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 7855–7862, 2019.
Crossref
[37]
Liu, L.; Li, H. D.; Dai, Y. C. Efficient global 2D–3D matching for camera localization in a large-scale 3D map. In: Proceedings of the IEEE International Conference on Computer Vision, 2391–2400, 2017.
Crossref
[38]

Sattler, T.; Leibe, B.; Kobbelt, L. Efficient & effective prioritized matching for large-scale image-based localization. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 39, No. 9, 1744–1756, 2017.
Crossref Google Scholar
[39]
Yu, C.; Liu, Z. X.; Liu, X. J.; Xie, F. G.; Yang, Y.; Wei, Q.; Fei, Q. DS-SLAM: A semantic visual SLAM towards dynamic environments. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 1168–1174, 2018.
Crossref
[40]
Runz, M.; Buffier, M.; Agapito, L. MaskFusion: Real-time recognition, tracking and reconstruction of multiple moving objects. In: Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, 10–20, 2018.
Crossref
[41]
Valada, A.; Radwan, N.; Burgard, W. Deep auxiliary learning for visual localization and odometry. In: Proceedings of the IEEE International Conference on Robotics and Automation, 6939–6946, 2018.
Crossref
[42]
Balntas, V.; Li, S. D.; Prisacariu, V. RelocNet: Continuous metric learning relocalisation using neural nets. In: Computer Vision – ECCV 2018. Lecture Notes in Computer Science, Vol. 11218. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 782–799, 2018.
Crossref
[43]
Melekhov, I.; Ylioinas, J.; Kannala, J.; Rahtu, E. Image-based localization using hourglass networks. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, 870–877, 2017.
Crossref
[44]
Laskar, Z.; Melekhov, I.; Kalia, S.; Kannala, J. Camera relocalization by computing pairwise relative poses using convolutional neural network. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, 920–929, 2017.
Crossref
[45]
Brachmann, E.; Rother, C. Learning less is more—6D camera localization via 3D surface regression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4654–4662, 2018.
Crossref
[46]
Wang, J. Y.; Qi, Y. Camera relocalization using deep point cloud generation and hand-crafted feature refinement. In: Proceedings of the IEEE International Conference on Robotics and Automation, 5891–5897, 2021.
Crossref
[47]
Brachmann, E.; Krull, A.; Nowozin, S.; Shotton, J.; Michel, F.; Gumhold, S.; Rother, C. DSAC—Differentiable RANSAC for camera localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2492–2500, 2017.
Crossref
[48]
Girshick, R.; Donahue, J.; Darrell, T.; Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 580–587, 2014.
Crossref
[49]
Girshick, R. Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, 1440–1448, 2015.
Crossref
[50]

Lan, Y. Q.; Duan, Y.; Liu, C. Y.; Zhu, C. Y.; Xiong, Y. S.; Huang, H.; Xu, K. ARM3D: Attention-based relation module for indoor 3D object detection. Computational Visual Media Vol. 8, No. 3, 395–414, 2022.
Crossref Google Scholar
[51]

Ren, S. Q.; He, K. M.; Girshick, R.; Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 39, No. 6, 1137–1149, 2017.
Crossref Google Scholar
[52]
He, K. M.; Gkioxari, G.; Dollar, P.; Girshick, R. Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, 2980–2988, 2017.
Crossref
[53]
Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C. Y.; Berg, A. C. SSD: Single shot MultiBox detector. In: Computer Vision – ECCV 2016. Lecture Notes in Computer Science, Vol. 9905. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 21–37, 2016.
Crossref
[54]
Jocher, G. Yolo v5. 2020. Available at https://github.com/ultralytics/yolov5
[55]
Dai, A.; Chang, A. X.; Savva, M.; Halber, M.; Funkhouser, T.; Niessner, M. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2432–2443, 2017.
Crossref
[56]
Charles, R. Q.; Hao, S.; Mo, K. C.; Guibas, L. J. PointNet: Deep learning on point sets for 3D classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 77–85, 2017.
Crossref
[57]
Tchapmi, L. P.; Kosaraju, V.; Rezatofighi, H.; Reid, I.; Savarese, S. TopNet: Structural point cloud decoder. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 383–392, 2019.
Crossref
[58]
Fan, H. Q.; Su, H.; Guibas, L. A point set generation network for 3D object reconstruction from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2463–2471, 2017.
Crossref
[59]

Huang, S. S.; Chen, H. X.; Huang, J. H.; Fu, H. B.; Hu, S. M. Real-time globally consistent 3D reconstruction with semantic priors. IEEE Transactions on Visualization and Computer Graphics Vol. 29, No. 4, 1977–1991, 2023.
Crossref Google Scholar
[60]

Zheng, T.; Zhang, G. Q.; Han, L.; Xu, L.; Fang, L. Building fusion: Semantic-aware structural building-scale 3D reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 44, No. 5, 2328–2345, 2022.
Google Scholar
[61]

Zou, Z. X.; Huang, S. S.; Mu, T. J.; Wang, Y. P. ObjectFusion: Accurate object-level SLAM with neural object priors. Graphical Models Vol. 123, 101165, 2022.
Crossref Google Scholar
[62]

Mur-Artal, R.; Montiel, J. M. M.; Tardos, J. D. ORB-SLAM: A versatile and accurate monocular SLAM system. IEEE Transactions on Robotics Vol. 31, No. 5, 1147–1163, 2015.
Crossref Google Scholar
[63]

Mur-Artal, R.; Tardós, J. D. ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Transactions on Robotics Vol. 33, No. 5, 1255–1262, 2017.
Crossref Google Scholar
[64]
Newcombe, R. A.; Izadi, S.; Hilliges, O.; Molyneaux, D.; Kim, D.; Davison, A. J.; Kohli, P.; Shotton, J.; Hodges, S.; Fitzgibbon, A. KinectFusion: Real-time dense surface mapping and tracking. In: Proceedings of the 10th IEEE International Symposium on Mixed and Augmented Reality, 127–136, 2011.
Crossref
[65]
Kingma, D. P.; Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[66]

Wang, B.; Chen, C. H.; Xiaoxuan Lu, C.; Zhao, P. J.; Trigoni, N.; Markham, A. AtLoc: Attention guided camera localization. Proceedings of the AAAI Conference on Artificial Intelligence Vol. 34, No. 6, 10393–10401, 2020.
Crossref Google Scholar
[67]

Taira, H.; Okutomi, M.; Sattler, T.; Cimpoi, M.; Pollefeys, M.; Sivic, J.; Pajdla, T.; Torii, A. InLoc: Indoor visual localization with dense matching and view synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 43, No. 4, 1293–1307, 2021.
Crossref Google Scholar
[68]
Yang, L. W.; Bai, Z. Q.; Tang, C. Z.; Li, H. H.; Furukawa, Y.; Tan, P. SANet: Scene agnostic network for camera localization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 42–51, 2019.
Crossref
[69]
Tang, S. T.; Tang, C. Z.; Huang, R.; Zhu, S. Y.; Tan, P. Learning camera localization via dense scene matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1831–1841, 2021.
Crossref
[70]
Li, X. T.; Wang, S. Z.; Zhao, Y.; Verbeek, J.; Kannala, J. Hierarchical scene coordinate classification and regression for visual localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11980–11989, 2020.
Crossref
[71]

Du, Z. J.; Huang, S. S.; Mu, T. J.; Zhao, Q. H.; Martin, R. R.; Xu, K. Accurate dynamic SLAM using CRF-based long-term consistency. IEEE Transactions on Visualization and Computer Graphics Vol. 28, No. 4, 1745–1757, 2022.
Crossref Google Scholar
Computational Visual Media
Volume 10 Issue 5,
October 2024
Pages 993-1011
DOI: 10.1007/s41095-022-0319-z
Cite this article:
Wang J, Qi Y. Multi-task learning and joint refinement between camera localization and object detection. Computational Visual Media, 2024, 10(5): 993-1011. https://doi.org/10.1007/s41095-022-0319-z

91

Views

2

Downloads

0

Crossref

0

Web of Science

0

Scopus

0

CSCD

Altmetrics
Received: 03 July 2022
Accepted: 03 October 2022
Published: 08 February 2024
© The Author(s) 2024.

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
Return