AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Journal of Computer Science and Technology Article
Article Link
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Regular Paper

Element-Arrangement Context Network for Facade Parsing

National Engineering Laboratory for Brain-Inspired Intelligence Technology and Application, University of Science and Technology of China, Hefei 230026, China
Show Author Information

Abstract

Facade parsing aims to decompose a building facade image into semantic regions of the facade objects. Considering each architectural element on a facade as a parameterized rectangle, we formulate the facade parsing task as object detection, allowing overlapping and nesting, which will support structural 3D modeling and editing for further applications. In contrast to general object detection, the spatial arrangement regularity and appearance similarity between the facade elements of the same category provide valuable context for accurate element localization. In this paper, we propose to exploit the spatial arrangement regularity and appearance similarity of facade elements in a detection framework. Our element-arrangement context network (EACNet) consists of two unidirectional attention branches, one to capture the column-context and the other to capture row-context to aggregate element-specific features from multiple instances on the facade. We conduct extensive experiments on four public datasets (ECP, CMP, Graz50, and eTRIMS). The proposed EACNet achieves the highest mIoU (82.1% on ECP, 77.35% on Graz50, and 82.3% on eTRIMS) compared with the state-of-the-art methods. Both the quantitative and qualitative evaluation results demonstrate the effectiveness of our dual unidirectional attention branches to parse facade elements.

Keywords

facade parsingelement detectionlayout regularityspatial context

Electronic Supplementary Material

Download File(s)
jcst-37-3-652-Highlights.pdf (694.1 KB)

References

[1]

Müller P, Zeng G, Wonka P, van Gool L. Image-based procedural modeling of facades. ACM Transactions on Graphics, 2007, 26(3): Article No. 85. DOI: 10.1145/1276377.1276484.
Crossref Google Scholar
[2]

Shen C H, Huang S S, Fu H B, Hu S M. Adaptive partitioning of urban facades. ACM Transactions on Graphics, 2011, 30(6): Article No. 184. DOI: 10.1145/2070781.2024218.
Crossref Google Scholar
[3]
Teboul O, Simon L, Koutsourakis P, Paragios N. Segmentation of building facades using procedural shape priors. In Proc. the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2010, pp. 3105-3112. DOI: 10.1109/CVPR.2010.5540068.
Crossref
[4]
Teboul O, Kokkinos I, Simon L, Koutsourakis P, Paragios N. Shape grammar parsing via reinforcement learning. In Proc. the 2011 IEEE Conference on Computer Vision and Pattern Recognition, June 2011, pp. 2273-2280. DOI: 10.1109/CVPR.2011.5995319.
Crossref
[5]
Yang C, Han T, Quan L, Tai C L. Parsing façade with rank-one approximation. In Proc. the 2012 IEEE Conference on Computer Vision and Pattern Recognition, June 2012, pp. 1720-1727. DOI: 10.1109/CVPR.2012.6247867.
Crossref
[6]
Cohen A, Schwing A G, Pollefeys M. Efficient structured parsing of facades using dynamic programming. In Proc. the 2014 IEEE Conference on Computer Vision and Pattern Recognition, June 2014, pp. 3206-3213. DOI: 10.1109/CVPR.2014.410.
Crossref
[7]
Martinović A, Mathias M, Weissenberg J, van Gool L. A three-layered approach to facade parsing. In Proc. the 12th European Conference on Computer Vision, Oct. 2012, pp. 416-429. DOI: 10.1007/978-3-642-33786-4_31.
Crossref
[8]

Mathias M, Martinović A, van Gool L. ATLAS: A three-layered approach to facade parsing. International Journal of Computer Vision, 2016, 118(1): 22-48. DOI: 10.1007/s11263-015-0868-z.
Crossref Google Scholar
[9]

Schmitz M, Mayer H. A convolutional network for semantic facade segmentation and interpolation. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2016, XLI-B3: 709-715. DOI: 10.5194/isprs-archives-XLI-B3-709-2016.
Crossref Google Scholar
[10]
Liu H, Zhang J, Hoi S C H. DeepFacade: A deep learning approach to facade parsing. In Proc. the 26th International Joint Conference on Artificial Intelligence, Aug. 2017, pp. 2301-2307. DOI: 10.24963/ijcai.2017/320.
Crossref
[11]

Liu H, Xu Y, Zhang J, Zhu J, Li Y, Hoi S C H. DeepFacade: A deep learning approach to facade parsing with symmetric loss. IEEE Transactions on Multimedia, 2020, 22(12): 3153-3165. DOI: 10.1109/TMM.2020.2971431.
Crossref Google Scholar
[12]
Riemenschneider H, Krispel U, Thaller W, Donoser M, Havemann S, Fellner D, Bischof H. Irregular lattices for complex shape grammar facade parsing. In Proc. the 2012 IEEE Conference on Computer Vision and Pattern Recognition, June 2012, pp. 1640-1647. DOI: 10.1109/CVPR.2012.6247857.
Crossref
[13]
Tyleček R, Šára R. Spatial pattern templates for recognition of objects with regular structure. In Proc. the 35th German Conference on Pattern Recognition, Sept. 2013, pp. 364-374. DOI: 10.1007/978-3-642-40602-7_39.
Crossref
[14]

Bao F, Schwarz M, Wonka P. Procedural facade variations from a single layout. ACM Transactions on Graphics, 2013, 32(1): Article No. 8. DOI: 10.1145/2421636.2421644.
Crossref Google Scholar
[15]

Dang M, Ceylan D, Neubert B, Pauly M. SAFE: Structure-aware facade editing. Computer Graphics Forum, 2014, 33(2): 83-93. DOI: 10.1111/cgf.12313.
Crossref Google Scholar
[16]

Ilčík M, Musialski P, Auzinger T, Wimmer M. Layer-based procedural design of façades. Computer Graphics Forum, 2015, 34(2): 205-216. DOI: 10.1111/cgf.12553.
Crossref Google Scholar
[17]
Han F, Zhu S C. Bottom-up/top-down image parsing by attribute graph grammar. In Proc. the 10th IEEE International Conference on Computer Vision, Oct. 2005, pp. 1778-1785. DOI: 10.1109/ICCV.2005.50.
Crossref
[18]

Talton J O, Lou Y, Lesser S, Duke J, Měch R, Koltun V. Metropolis procedural modeling. ACM Transactions on Graphics, 2011, 30(2): Article No. 11. DOI: 10.1145/1944846.1944851.
Crossref Google Scholar
[19]

Yeh Y T, Breeden K, Yang L, Fisher M, Hanrahan P. Synthesis of tiled patterns using factor graphs. ACM Transactions on Graphics, 2013, 32(1): Article No. 3. DOI: 10.1145/2421636.2421639.
Crossref Google Scholar
[20]

Rahmani K, Huang H, Mayer H. Facade segmentation with a structured random forest. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, 2017, IV-1/W1: 175-181. DOI: 10.5194/isprs-annals-IV-1-W1-175-2017.
Crossref Google Scholar
[21]

Gaddle R, Jampani V, Marlet R, V Gehler P. Efficient 2D and 3D facade segmentation using auto-context. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(5): 1273-1280. DOI: 10.1109/TPAMI.2017.2696526.
Crossref Google Scholar
[22]
Koziński M, Gadde R, Zagoruyko S, Obozinski G, Marlet R. A MRF shape prior for facade parsing with occlusions. In Proc. the 2015 IEEE Conference on Computer Vision and Pattern Recognition, June 2015, pp. 2820-2828. DOI: 10.1109/CVPR.2015.7298899.
Crossref
[23]
Cohen A, R Oswald M, Liu Y, Pollefeys M. Symmetry-aware façade parsing with occlusions. In Proc. the 2017 International Conference on 3D Vision, Oct. 2017, pp. 393-401. DOI: 10.1109/3DV.2017.00052.
Crossref
[24]

Nan L, Sharf A, Zhang H, Cohen-Or D, Chen B. SmartBoxes for interactive urban reconstruction. ACM Trans. Graph. , 2010, 29(4): Article No. 93. DOI: 10.1145/1778765.1778830.
Crossref Google Scholar
[25]

Zhang H, Xu K, Jiang W, Lin J, Cohen-Or D, Chen B. Layered analysis of irregular facades via symmetry maximization. ACM Trans. Graph. , 2013, 32(4): Article No. 121. DOI: 10.1145/2461912.2461923.
Crossref Google Scholar
[26]
Femiani J, Reyaz Para W, Mitra N, Wonka P. Facade segmentation in the wild. arXiv: 1805.08634, 2018. https://arxiv.org/pdf/1805.08634.pdf, Jan. 2022.
[27]
He K, Gkioxari G, Dollár P, Girshick R. Mask RCNN. In Proc. the 2017 IEEE International Conference on Computer Vision, Oct. 2017, pp. 2961-2969. DOI: 10.1109/ICCV.2017.322.
Crossref
[28]
Girshick R, Donahue J, Darrell T, Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proc. the 2014 IEEE Conference on Computer Vision and Pattern Recognition, June 2014, pp. 580-587. DOI: 10.1109/CVPR.2014.81.
Crossref
[29]
Girshick R. Fast R-CNN. In Proc. the 2015 IEEE International Conference on Computer Vision, Dec. 2015, pp. 1440-1448. DOI: 10.1109/ICCV.2015.169.
Crossref
[30]
Law H, Deng J. CornerNet: Detecting objects as paired keypoints. In Proc. the 15th European Conference on Computer Vision, Sept. 2018, pp. 734-750. DOI: 10.1007/978-3-030-01264-9_45.
Crossref
[31]
Newell A, Huang Z, Deng J. Associative embedding: End-to-end learning for joint detection and grouping. In Proc. the Annual Conference onNeural Information Processing Systems, Dec. 2017, pp. 2277-2287.
[32]
Zhou X, Wang D, Krähenbühl P. Objects as points. arXiv: 1904.07850, 2019. https://arxiv.org/pdf/1904.07850.pdf, Jan. 2022.
[33]
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser L, Polosukhin I. Attention is all you need. In Proc. the Annual Conference on Neural Information Processing Systems, Dec. 2017, pp. 5998-6008.
[34]
Yang Z, Yang D, Dyer C, He X, Smola A, Hovy E. Hierarchical attention networks for document classification. In Proc. the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2016, pp. 1480-1489. DOI: 10.18653/v1/N16-1174.
Crossref
[35]

Roy A, Saffar M, Vaswani A, Grangier D. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 2021, 9: 53-68. DOI: 10.1162/tacl_a_00353.
Crossref Google Scholar
[36]
Sarlin P E, DeTone D, Malisiewicz T, Rabinovich A. SuperGlue: Learning feature matching with graph neural networks. In Proc. the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2020, pp. 4938-4947. DOI: 10.1109/CVPR42600.2020.00499.
Crossref
[37]
Kolesnikov A, Dosovitskiy A, Weissenborn D, Heigold G, Uszkoreit J, Beyer L, Minderer M, Dehghani M, Houlsby N, Gelly S, Unterthiner T, Zhai X. An image is worth 16x16 words: Transformers for image recognition at scale. In Proc. the 9th International Conference on Learning Representations, May 2021.
[38]
Wang S, Li B Z, Khabsa M, Fang H, Ma H. Linformer: Self-attention with linear complexity. arXiv: 2006.04768, 2020. https://arxiv.org/pdf/2006.04768.pdf, Jan. 2022.
[39]
Hu J, Shen L, Sun G. Squeeze-and-excitation networks. In Proc. the 2018 IEEE Conference on Computer Vision and Pattern Recognition, June 2018, pp. 7132-7141. DOI: 10.1109/CVPR.2018.00745.
Crossref
[40]
Zhao H, Zhang Y, Liu S, Shi J, Loy C C, Lin D, Jia J. PSANet: Point-wise spatial attention network for scene parsing. In Proc. the 15th European Conference on Computer Vision, Sept. 2018, pp. 267-283. DOI: 10.1007/978-3-030-01240-3_17.
Crossref
[41]
Wang X, Girshick R, Gupta A, He K. Non-local neural networks. In Proc. the 2018 IEEE Conference on Computer Vision and Pattern Recognition, June 2018, pp. 7794-7803. DOI: 10.1109/CVPR.2018.00813.
Crossref
[42]
Strudel R, Garcia R, Laptev I, Schmid C. Segmenter: Transformer for semantic segmentation. In Proc. the 2021 IEEE/CVF International Conference on Computer Vision, Oct. 2021, pp. 7262-7272. DOI: 10.1109/ICCV48922.2021.00717.
Crossref
[43]
Wang W, Xie E, Li X, Fan D P, Song K, Liang D, Lu T, Luo P, Shao L. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proc. the 2021 IEEE/CVF International Conference on Computer Vision, Oct. 2021, pp. 568-578. DOI: 10.1109/ICCV48922.2021.00061.
Crossref
[44]
Huang Z, Wang X, Wei Y, Huang L, Shi H, Liu W, S Huang T. CCNet: Criss-cross attention for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. DOI: 10.1109/TPAMI.2020.3007032.
Crossref
[45]
Newell A, Yang K, Deng J. Stacked hourglass networks for human pose estimation. In Proc. the 14th European Conference on Computer Vision, Oct. 2016, pp. 483-499. DOI: 10.1007/978-3-319-46484-8_29.
Crossref
[46]
Lin T Y, Goyal P, Girshick R, He K, Dollár P. Focal loss for dense object detection. In Proc. the 2017 IEEE International Conference on Computer Vision, Oct. 2017, pp. 2999-3007. DOI: 10.1109/ICCV.2017.324.
Crossref
[47]
Koziński M, Obozinski G, Marlet R. Beyond procedural facade parsing: Bidirectional alignment via linear programming. In Proc. the 12th Asian Conference on Computer Vision, Nov. 2015, pp. 79-94. DOI: 10.1007/978-3-319-16817-3_6.
Crossref
[48]

Rahmani K, Huang H, Mayer H. High quality facade segmentation base on structured random forest, region proposal network and rectangular fitting. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, 2018, IV-2: 223-230. DOI: 10.5194/isprs-annals-IV-2-223-2018.
Crossref Google Scholar
[49]
Lin T Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick C L. Microsoft COCO: Common objects in context. In Proc. the 13th European Conference on Computer Vision, Sept. 2014, pp. 740-755. DOI: 10.1007/978-3-319-10602-1_48.
Crossref
[50]

Zhang Z, Ganesh A, Liang X, Ma Y. TILT: Transform invariant low-rank textures. International Journal of Computer Vision, 2012, 99(1): 1-24. DOI: 10.1007/s11263-012-0515-x.
Crossref Google Scholar
[51]
He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp. 770-778. DOI: 10.1109/CVPR.2016.90.
Crossref
[52]
Yu F, Wang D, Shelhamer E, Darrell T. Deep layer aggregation. In Proc. the 2018 IEEE Conference on Computer Vision and Pattern Recognition, June 2018, pp. 2403-2412. DOI: 10.1109/CVPR.2018.00255.
Crossref
Journal of Computer Science and Technology
Volume 37 Issue 3,
May 2022
Pages 652-665
DOI: 10.1007/s11390-022-2189-3
Cite this article:
Tao Y, Zhang Y-T, Chen X-J. Element-Arrangement Context Network for Facade Parsing. Journal of Computer Science and Technology, 2022, 37(3): 652-665. https://doi.org/10.1007/s11390-022-2189-3

388

Views

2

Crossref

2

Web of Science

2

Scopus

0

CSCD

Altmetrics
Received: 28 January 2022
Accepted: 24 April 2022
Published: 31 May 2022
©Institute of Computing Technology, Chinese Academy of Sciences 2022
Return