Home Journal of Computer Science and Technology Article
Article Link
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript
Show Outline
Outline
Abstract
Keywords
Electronic Supplementary Material
References
Show full outline
Hide outline
Regular Paper

ScenePalette: Contextually Exploring Object Collections Through Multiplex Relations in 3D Scenes

Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
Beijing National Research Center for Information Science and Technology, Beijing 100084, China
Show Author Information

Abstract

This paper presents ScenePalette, a modeling tool that allows users to “draw” 3D scenes interactively by placing objects on a canvas based on their contextual relationship. ScenePalette is inspired by an important intuition which was often ignored in previous work: a real-world 3D scene consists of the contextually reasonable organization of objects, e.g. people typically place one double bed with several subordinate objects into a bedroom instead of different shapes of beds. ScenePalette, abstracts 3D repositories as multiplex networks and accordingly encodes implicit relations between or among objects. Specifically, basic statistics such as co-occurrence, in combination with advanced relations, are used to tackle object relationships of different levels. Extensive experiments demonstrate that the latent space of ScenePalette has rich contexts that are essential for contextual representation and exploration.

Keywords

computer graphics3D scene context3D repository explorationmultiplex network embedding

Electronic Supplementary Material

Download File(s)
JCST-2201-12194-Highlights.pdf (281.4 KB)

References

[1]
Song S, Yu F, Zeng A, Chang A X, Savva M, Funkhouser T. Semantic scene completion from a single depth image. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp.190–198. DOI: 10.1109/CVPR.2017.28.
Crossref
[2]
Fu H, Cai B, Gao L, Zhang L X, Wang J, Li C, Zeng Q, Sun C, Jia R, Zhao B, Zhang H. 3D-FRONT: 3D furnished rooms with layOuts and semaNTics. In Proc. the 2021 IEEE/CVF International Conference on Computer Vision, Oct. 2021, pp.10913–10922. DOI: 10.1109/ICCV48922.2021.01075.
Crossref
[3]

Fu Q, Chen X, Wang X, Wen S, Zhou B, Fu H. Adaptive synthesis of indoor scenes via activity-associated object relation graphs. ACM Trans. Graphics, 2017, 36(6): Article No. 201. DOI: 10.1145/3130800.3130805.

Crossref Google Scholar
[4]

Zhang S H, Zhang S K, Liang Y, Hall P. A survey of 3D indoor scene synthesis. Journal of Computer Science and Technology, 2019, 34(3): 594–608. DOI: 10.1007/s11390-019-1929-5.

Crossref Google Scholar
[5]

Zhang S H, Zhang S K, Xie W Y, Luo C Y, Yang Y L, Fu H. Fast 3D indoor scene synthesis by learning spatial relation priors of objects. IEEE Trans. Visualization and Computer Graphics, 2022, 28(9): 3082–3092. DOI: 10.1109/TVCG.2021.3050143.

Crossref Google Scholar
[6]
Yan M, Chen X, Zhou J. An interactive system for efficient 3D furniture arrangement. In Proc. the 2017 Computer Graphics International Conference, Jun. 2017, Article No. 29. DOI: 10.1145/3095140.3095169.
Crossref
[7]
Zhang S K, Li Y X, He Y, Yang Y L, Zhang S H. MageAdd: Real-time interaction simulation for scene synthesis. In Proc. the 29th ACM International Conference on Multimedia, Oct. 2021, pp.965–973. DOI: 10.1145/3474085.3475194.
Crossref
[8]
Handa A, Patraucean V, Badrinarayanan V, Stent S, Cipolla R. Understanding real world indoor scenes with synthetic data. In Proc. the 2016 IEEE Conference on Computer Vision, Jun. 2016, pp.4077–4085. DOI: 10.1109/CVPR.2016.442.
Crossref
[9]
Luo A, Zhang Z, Wu J, Tenenbaum J B. End-to-end optimization of scene layout. In Proc. the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2020, pp.3753–3762. DOI: 10.1109/CVPR42600.2020.00381.
Crossref
[10]

Huang S S, Shamir A, Shen C H, Zhang H, Sheffer A, Hu S M, Cohen-Or D. Qualitative organization of collections of shapes via quartet analysis. ACM Trans. Graphics, 2013, 32(4): Article No. 71. DOI: 10.1145/2461912.2461954.

Crossref Google Scholar
[11]

Chen D Y, Tian X P, Shen Y T, Ouhyoung M. On visual similarity based 3D model retrieval. Computer Graphics Forum, 2003, 22(3): 223–232. DOI: 10.1111/1467-8659.00669.

Crossref Google Scholar
[12]

Cai H Y, Zheng V W, Chang K C C. A comprehensive survey of graph embedding: Problems, techniques, and applications. IEEE Trans. Knowledge and Data Engineering, 2018, 30(9): 1616–1637. DOI: 10.1109/TKDE.2018.2807452.

Crossref Google Scholar
[13]
Zhang H, Qiu L, Yi L, Song Y. Scalable multiplex network embedding. In Proc. the 27th International Joint Conference on Artificial Intelligence, Jul. 2018, pp.3082–3088. DOI: 10.5555/3304889.3305089.
Crossref
[14]

Zhang S K, Xie W Y, Zhang S H. Geometry-based layout generation with hyper-relations AMONG objects. Graphical Models, 2021, 116: 101104. DOI: 10.1016/j.gmod.2021.101104.

Crossref Google Scholar
[15]

He Y, Shen Z, Cui P. Towards Non-I. I. D. image classification: A dataset and baselines. Pattern Recognition, 2021, 110: 107383. DOI: 10.1016/j.patcog.2020.107383.

Crossref Google Scholar
[16]

Yu L F, Yeung S K, Tang C K, Terzopoulos D, Chan T F, Osher S. Make it home: Automatic optimization of furniture arrangement. ACM Trans. Graphics, 2011, 30(4): 86. DOI: 10.1145/2010324.1964981.

Crossref Google Scholar
[17]
Chang A, Savva M, Manning C D. Learning spatial knowledge for text to 3D scene generation. In Proc. the 2014 Conference on Empirical Methods in Natural Language Processing, Oct. 2014, pp.2028–2038. DOI: 10.3115/v1/D14-1217.
Crossref
[18]
Perozzi B, Al-Rfou R, Skiena S. DeepWalk: Online learning of social representations. In Proc. the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Aug. 2014, pp.701–710. DOI: 10.1145/2623330.2623732.
Crossref
[19]
He R, McAuley J. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proc. the 25th International Conference on World Wide Web, Apr. 2016, pp.507–517. DOI: 10.1145/2872427.2883037.
Crossref
[20]
Tang L, Wang X, Liu H. Uncoverning groups via heterogeneous interaction analysis. In Proc. the 9th IEEE International Conference on Data Mining, Dec. 2009, pp.503–512. DOI: 10.1109/ICDM.2009.20.
Crossref
[21]

Fisher M, Hanrahan P. Context-based search for 3D models. ACM Trans. Graphics, 2010, 29(6): Article No. 182. DOI: 10.1145/1882261.1866204.

Crossref Google Scholar
[22]

Xu K, Chen K, Fu H, Sun W L, Hu S M. Sketch2Scene: Sketch-based co-retrieval and co-placement of 3D models. ACM Trans. Graphics, 2013, 32(4): Article No. 123. DOI: 10.1145/2461912.2461968.

Crossref Google Scholar
[23]

Weiss T, Litteneker A, Duncan N, Nakada M, Jiang C, Yu L F, Terzopoulos D. Fast and scalable position-based layout synthesis. IEEE Trans. Visualization and Computer Graphics, 2019, 25(12): 3231–3243. DOI: 10.1109/TVCG.2018.2866436.

Crossref Google Scholar
[24]

Fisher M, Savva M, Hanrahan P. Characterizing structural relationships in scenes using graph kernels. ACM Trans. Graphics, 2011, 30(4): Article No. 34. DOI: 10.1145/2010324.1964929.

Crossref Google Scholar
[25]

Xu K, Ma R, Zhang H, Zhu C, Shamir A, Cohen-Or D, Huang H. Organizing heterogeneous scene collections through contextual focal points. ACM Trans. Graphics, 2014, 33(4): Article No. 35. DOI: 10.1145/2601097.2601109.

Crossref Google Scholar
[26]

Cui P, Wang X, Pei J, Zhu W. A survey on network embedding. IEEE Trans. Knowledge and Data Engineering, 2019, 31(5): 833–852. DOI: 10.1109/TKDE.2018.2849727.

Crossref Google Scholar
[27]
Wang X, Cui P, Wang J, Pei J, Zhu W, Yang S. Community preserving network embedding. In Proc. the 31st AAAI Conference on Artificial Intelligence, Nov. 2017, pp.203–209. DOI: 10.1145/3357384.3357947.
Crossref
[28]
Newell A, Huang Z, Deng J. Associative embedding: End-to-end learning for joint detection and grouping. In Proc. the 31st International Conference on Neural Information Processing Systems, Dec. 2017, pp.2274–2284.
[29]

Kleiman Y, van Kaick O, Sorkine-Hornung O, Cohen-Or D. SHED: Shape edit distance for fine-grained shape similarity. ACM Trans. Graphics, 2015, 34(6): Article No. 235. DOI: 10.1145/2816795.2818116.

Crossref Google Scholar
[30]

Kohonen T. Self-organized formation of topologically correct feature maps. Biological Cybernetics, 1982, 43(1): 59–69. DOI: 10.5555/65669.104428.

Crossref Google Scholar
[31]
Liu W, Chen P Y, Yeung S, Suzumura T, Chen L. Principled multilayer network embedding. In Proc. the 2017 IEEE International Conference on Data Mining Workshops, Nov. 2017, pp.134–141. DOI: 10.1109/ICDMW.2017.23.
Crossref
[32]
De Sá H R, Prudêncio R B C. Supervised link prediction in weighted networks. In Proc. the 2011 International Joint Conference on Neural Networks, Sept. 2011, pp.2281–2288. DOI: 10.1109/IJCNN.2011.6033513.
Crossref
[33]

Tangelder J W H, Veltkamp R C. A survey of content based 3D shape retrieval methods. Multimedia Tools and Applications, 2008, 39(3): 441–471. DOI: 10.1007/s11042-007-0181-0.

Crossref Google Scholar
[34]
Kazhdan M, Funkhouser T, Rusinkiewicz S. Rotation invariant spherical harmonic representation of 3D shape descriptors. In Proc. the 2003 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing, Jun. 2003, pp.156–164. DOI: 10.5555/882370.882392.
[35]

Osada R, Funkhouser T, Chazelle B, Dobkin D. Shape distributions. ACM Trans. Graphics, 2002, 21(4): 807–832. DOI: 10.1145/571647.571648.

Crossref Google Scholar
[36]
Shilane P, Min P, Kazhdan M, Funkhouser T. The princeton shape benchmark. In Proc. the 2004 Shape Modeling Applications, Jun. 2004, pp.167–178. DOI: 10.1109/SMI.2004.1314504.
Crossref
[37]
Charles R Q, Su H, Kaichun M, Guibas L J. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp.77–85. DOI: 10.1109/CVPR.2017.16.
Crossref
[38]
Zeng A, Song S, Nießner M, Fisher M, Xiao J, Funkhouser T. 3DMatch: Learning local geometric descriptors from RGB-D reconstructions. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp.199–208. DOI: 10.1109/CVPR.2017.29.
Crossref
[39]
Chang A X, Funkhouser T, Guibas L, Hanrahan P, Huang Q, Li Z, Savarese S, Savva M, Song S, Su H, Xiao J, Yi L, Yu F. ShapeNet: An information-rich 3D model repository. arXiv: 1512.03012, 2015. https://arxiv.org/abs/1512.03012, Sept. 2024.
[40]

Diggle P J, Besag J, Gleaves J T. Statistical analysis of spatial point patterns by means of distance methods. Biometrics, 1976, 32(3): 659–667. DOI: 10.2307/2529754.

Crossref Google Scholar
[41]

Gignoux J, Duby C, Barot S. Comparing the performances of Diggle’s tests of spatial randomness for small samples with and without edge-effect correction: Application to ecological data. Biometrics, 1999, 55(1): 156–164. DOI: 10.1111/j.0006-341x.1999.00156.x.

Crossref Google Scholar
[42]
Rosin P. Thresholding for change detection. In Proc. the 6th International Conference on Computer Vision, Jan. 1998, pp.274–279. DOI: 10.1109/ICCV.1998.710730.
Crossref
[43]

Diggle P J. On parameter estimation and goodness-of-fit testing for spatial point patterns. Biometrics, 1979, 35(1): 87–101. DOI: 10.2307/2529938.

Crossref Google Scholar
[44]

Assunção R. Testing spatial randomness by means of angles. Biometrics, 1994, 50(2): 531–537. DOI: 10.2307/2533397.

Crossref Google Scholar
[45]

van Kaick O, Fish N, Kleiman Y, Asafi S, Cohen-Or D. Shape segmentation by approximate convexity analysis. ACM Trans. Graphics, 2014, 34(1): Article No. 4. DOI: 10.1145/2611811.

Crossref Google Scholar
[46]

Hu M K. Visual pattern recognition by moment invariants. IRE Trans. Information Theory, 1962, 8(2): 179–187. DOI: 10.1109/TIT.1962.1057692.

Crossref Google Scholar
[47]
Gallager R G. Stochastic Processes: Theory for Applications. Cambridge University Press, 2013.
Crossref
[48]
Li J, Chen C, Tong H, Liu H. Multi-layered network embedding. In Proc. the 2018 SIAM International Conference on Data Mining, May 2018, pp.684–692. DOI: 10.1137/1.9781611975321.77.
Crossref
[49]

van der Maaten L, Hinton G. Visualizing data using t-SNE. Journal of Machine Learning Research, 2008, 9(86): 2579–2605.

Google Scholar
[50]

Yu L F, Yeung S K, Terzopoulos D. The clutterpalette: An interactive tool for detailing indoor scenes. IEEE Trans. Visualization and Computer Graphics, 2016, 22(2): 1138–1148. DOI: 10.1109/TVCG.2015.2417575.

Crossref Google Scholar
Journal of Computer Science and Technology
Volume 39 Issue 5,
September 2024
Pages 1180-1192
DOI: 10.1007/s11390-022-2194-6
Cite this article:
Zhang S-K, Xie W-Y, Wang C, et al. ScenePalette: Contextually Exploring Object Collections Through Multiplex Relations in 3D Scenes. Journal of Computer Science and Technology, 2024, 39(5): 1180-1192. https://doi.org/10.1007/s11390-022-2194-6
Metrics & Citations  
Article History
Copyright
Return