AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Computational Visual Media Article
PDF (4.5 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Research Article | Open Access

Neighborhood co-occurrence modeling in 3D point cloud segmentation

Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
Shanghai CLS Fintech Co., LTD, Shanghai 200030, China
MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai 200240, China
Show Author Information

Graphical Abstract

Abstract

A significant performance boost has been achieved in point cloud semantic segmentation by utilization of the encoder-decoder architecture and novel convolution operations for point clouds. However, co-occurrence relationships within a local region which can directly influence segmentation results are usually ignored by current works. In this paper, we propose a neighborhood co-occurrence matrix (NCM) to model local co-occurrence relationships in a point cloud. Wegenerate target NCM and prediction NCM fromsemantic labels and a prediction map respectively. Then,Kullback-Leibler (KL) divergence is used to maximize the similarity between the target and prediction NCMs to learn the co-occurrence relationship. Moreover, for large scenes where the NCMs for a sampled point cloud and the whole scene differ greatly, we introduce a reverse form of KL divergence which can better handle the difference to supervise the prediction NCMs. We integrate our method into an existing backbone and conduct comprehensive experiments on three datasets: Semantic3D for outdoor space segmentation, and S3DIS and ScanNet v2 for indoor scene segmentation. Results indicate that our method can significantly improve upon the backbone and outperform many leading competitors.

Keywords

semantic segmentation3D visionpoint cloudco-occurrence relationmodeling

References

[1]
Verdoja, F.; Thomas, D.; Sugimoto, A. Fast 3D point cloud segmentation using supervoxels with geometry and color for 3D scene understanding. In: Proceedings of the IEEE International Conference on Multimedia and Expo, 1285-1290, 2017.
Crossref
[2]
Xu, J. C.; Gong, J. Y.; Zhou, J.; Tan, X.; Xie, Y.; Ma, L. Z. SceneEncoder: Scene-aware semantic segmentation of point clouds with a learnable scene descriptor. In: Proceedings of the 29th International Joint Conference on Artificial Intelligence, 601-607, 2020.
[3]
Charles, R. Q.; Hao, S.; Mo, K. C.; Guibas, L. J. PointNet: Deep learning on point sets for 3D classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 77-85, 2017.
Crossref
[4]
Wu, W. X.; Qi, Z.; Fuxin, L. PointConv: Deep convolutional networks on 3D point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9613-9622, 2019.
[5]
Thomas, H.; Qi, C. R.; Deschaud, J. E.; Marcotegui, B.; Goulette, F.; Guibas, L. KPConv: Flexible and deformable convolution for point clouds. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 6410-6419, 2019.
Crossref
[6]
Hu, S.-M.; Cai, J.-X.; Lai, Y.-K. Semantic labeling and instance segmentation of 3D point clouds using patch context analysis and multiscale processing. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 7, 2485-2498, 2020.
Crossref Google Scholar
[7]
Qi, C. R.; Yi, L.; Su, H.; Guibas, L. J. PointNet++: Deep hierarchical feature learning on point sets in a metric space. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, 5105-5114, 2017.
[8]
Wang, L.; Huang, Y. C.; Hou, Y. L.; Zhang, S. M.; Shan, J. Graph attention convolution for point cloud semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10288-10297, 2019.
Crossref
[9]
Hu, Q. Y.; Yang, B.; Xie, L. H.; Rosa, S.; Guo, Y. L.; Wang, Z. H.; Trigoni, N.; Markham, A. RandLA-net: Efficient semantic segmentation of large-scale point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11105-11114, 2020.
[10]
Zhao, L.; Tao, W. B. JSNet: Joint instance and semantic segmentation of 3D point clouds. Proceedings of the AAAI Conference on Artificial Intelligence Vol. 34, No. 7, 12951-12958, 2020.
Crossref Google Scholar
[11]
Pham, Q. H.; Nguyen, T.; Hua, B. S.; Roig, G.; Yeung, S. K. JSIS3D: Joint semantic-instance segmentation of 3D point clouds with multi-task pointwise networks and multi-value conditional random fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8819-8828, 2019.
Crossref
[12]
Hu, Z.; Zhen, M.; Bai, X.; Fu, H.; Tai, C. JSENet: Joint semantic segmentation and edge detection network for 3D point clouds. In: Computer Vision-ECCV 2020. Lecture Notes in Computer Science, Vol. 12365. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 222-239, 2020.
[13]
Gong, J.; Xu, J.; Tan, X.; Zhou, J.; Qu, Y.; Xie, Y.; Ma, L. Boundary-aware geometric encoding for semantic segmentation of point clouds. In: Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
[14]
Zhang, J. Z.; Zhu, C. Y.; Zheng, L. T.; Xu, K. Fusion-aware point convolution for online semantic 3D scene segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4533-4542, 2020.
Crossref
[15]
Mattausch, O.; Panozzo, D.; Mura, C.; Sorkine-Hornung, O.; Pajarola, R. Object detection and classification from large-scale cluttered indoor scans. Computer Graphics Forum Vol. 33, No. 2, 11-21, 2014.
Crossref Google Scholar
[16]
Mottaghi, R.; Chen, X. J.; Liu, X. B.; Cho, N. G.; Lee, S. W.; Fidler, S.; Urtasun, R.; Yuille, A. The role of context for object detection and semantic segmentation in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 891-898, 2014.
Crossref
[17]
Zhao, S.; Wang, Y.; Yang, Z.; Cai, D. Region mutual information loss for semantic segmentation. In: Proceedings of the 33rd Conference on Neural Information Processing Systems, 11117-11127, 2019.
[18]
Wu, B. C.; Wan, A.; Yue, X. Y.; Keutzer, K. SqueezeSeg: Convolutional neural nets with recurrent CRF for real-time road-object segmentation from 3D LiDAR point cloud. In: Proceedings of the IEEE International Conference on Robotics and Automation, 1887-1893, 2018.
[19]
Ye, X. Q.; Li, J. M.; Huang, H. X.; Du, L.; Zhang, X. L. 3D recurrent neural networks with context fusion for point cloud semantic segmentation. In: Computer Vision-ECCV 2018. Lecture Notes in Computer Science, Vol. 11211. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 415-430, 2018.
[20]
Jiang, L.; Zhao, H. S.; Liu, S.; Shen, X. Y.; Fu, C. W.; Jia, J. Y. Hierarchical point-edge interaction network for point cloud semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 10432-10440, 2019.
Crossref
[21]
Zhang, H.; Zhang, H.; Wang, C. G.; Xie, J. Y. Co-occurrent features in semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 548-557, 2019.
Crossref
[22]
Deng, Z.; Todorovic, S.; Latecki, L. J. Semantic segmentation of RGBD images with mutex constraints. In: Proceedings of the IEEE International Conference on Computer Vision, 1733-1741, 2015.
Crossref
[23]
Koppula, H. S.; Anand, A.; Joachims, T.; Saxena, A. Semantic labeling of 3D point clouds for indoor scenes. In: Proceedings of the 24th International Conference on Neural Information Processing Systems, 244-252, 2011.
[24]
Zhao, Z.; Liu, T.; Li, S.; Li, B. F.; Du, X. Y. Ngram2vec: Learning improved word representations from ngram co-occurrence statistics. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, 244-253, 2017.
Crossref
[25]
Wang, Y. S.; Ma, X. J.; Chen, Z. Y.; Luo, Y.; Yi, J. F.; Bailey, J. Symmetric cross entropy for robust learning with noisy labels. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 322-330, 2019.
Crossref
[26]
Hackel, T.; Savinov, N.; Ladicky, L.; Wegner, J. D.; Schindler, K.; Pollefeys, M. Semantic3D.net: A new large-scale point cloud classification benchmark. arXiv preprint arXiv:1704.03847, 2017.
Crossref Google Scholar
[27]
Armeni, I.; Sener, O.; Zamir, A. R.; Jiang, H.; Brilakis, I.; Fischer, M.; Savarese, S. 3D semantic parsing of large-scale indoor spaces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1534-1543, 2016.
Crossref
[28]
Dai, A.; Chang, A. X.; Savva, M.; Halber, M.; Funkhouser, T.; Nießner, M. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2432-2443, 2017.
Crossref
[29]
Gong, J.; Xu, J.; Tan, X.; Song, H.; Qu, Y.; Xie, Y.; Ma, L. Omni-supervised point cloud segmentation via gradual receptive field component reasoning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11673-11682, 2021.
Crossref
[30]
Tchapmi, L.; Choy, C.; Armeni, I.; Gwak, J.; Savarese, S. SEGCloud: Semantic segmentation of 3D point clouds. In: Proceedings of the International Conference on 3D Vision, 537-547, 2017.
Crossref
[31]
Thomas, H.; Goulette, F.; Deschaud, J. E.; Marcotegui, B.; LeGall, Y. Semantic classification of 3D point clouds with multiscale spherical neighborhoods. In: Proceedings of the International Conference on 3D Vision, 390-398, 2018.
Crossref
[32]
Landrieu, L.; Simonovsky, M. Large-scale point cloud semantic segmentation with superpoint graphs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4558-4567, 2018.
Crossref
[33]
Zhang, Z. Y.; Hua, B. S.; Yeung, S. K. ShellNet: Efficient point cloud convolutional neural networks using concentric shells statistics. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 1607-1616, 2019.
Crossref
[34]
Khan, S. A.; Shi, Y. L.; Shahzad, M.; Zhu, X. X. FGCN: Deep feature-based graph convolutional network for semantic segmentation of urban 3D point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 778-787, 2020.
Crossref
[35]
Ma, Y. N.; Guo, Y. L.; Liu, H.; Lei, Y. J.; Wen, G. J. Global context reasoning for semantic segmentation of 3D point clouds. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 2920-2929, 2020.
[36]
Lei, H.; Akhtar, N.; Mian, A. SegGCN: Efficient 3D point cloud segmentation with fuzzy spherical kernel. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11608-11617, 2020.
Crossref
[37]
Huang, Q. G.; Wang, W. Y.; Neumann, U. Recurrent slice networks for 3D segmentation of point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2626-2635, 2018.
Crossref
[38]
Lin, Y. Q.; Yan, Z. Z.; Huang, H. B.; Du, D.; Liu, L. G.; Cui, S. G.; Han, X. FPConv: Learning local flattening for point convolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4292-4301, 2020.
Crossref
[39]
Han, W. K.; Wen, C. L.; Wang, C.; Li, X.; Li, Q. Point2Node: Correlation learning of dynamic-node for point cloud feature modeling. Proceedings of the AAAI Conference on Artificial Intelligence Vol. 34, No. 7, 10925-10932, 2020.
Crossref Google Scholar
[40]
Schult, J.; Engelmann, F.; Kontogianni, T.; Leibe, B. DualConvMesh-net: Joint geodesic and Euclidean convolutions on 3D meshes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8609-8619, 2020.
Crossref
[41]
Zhang, F. H.; Fang, J.; Wah, B.; Torr, P. Deep FusionNet for point cloud semantic segmentation. In: Computer Vision-ECCV 2020. Lecture Notes in Computer Science, Vol. 12369. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 644-663, 2020.
[42]
Li, Y. Y.; Bu, R.; Sun, M. C.; Wu, W.; Di, X. H.; Chen, B. Q. PointCNN: Convolution on X-transformed points. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, 828-838, 2018.
[43]
Huang, J. W.; Zhang, H. T.; Yi, L.; Funkhouser, T.; Nießner, M.; Guibas, L. J. TextureNet: Consistent local parametrizations for learning from high-resolution signals on meshes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4435-4444, 2019.
Crossref
[44]
Lei, H.; Akhtar, N.; Mian, A. Spherical kernel for efficient graph convolution on 3D point clouds. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 43, No. 10, 3664-3680, 2021.
Crossref Google Scholar
[45]
Yan, X.; Zheng, C. D.; Li, Z.; Wang, S.; Cui, S. G. PointASNL: Robust point clouds processing using nonlocal neural networks with adaptive sampling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5588-5597, 2020.
Crossref
Computational Visual Media
Volume 8 Issue 2,
June 2022
Pages 303-315
DOI: 10.1007/s41095-021-0244-6
Cite this article:
Gong J, Ye Z, Ma L. Neighborhood co-occurrence modeling in 3D point cloud segmentation. Computational Visual Media, 2022, 8(2): 303-315. https://doi.org/10.1007/s41095-021-0244-6

636

Views

32

Downloads

7

Crossref

7

Web of Science

7

Scopus

0

CSCD

Altmetrics
Received: 01 April 2021
Accepted: 28 May 2021
Published: 06 December 2021
© The Author(s) 2021.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www. editorialmanager.com/cvmj.
Return