Delving into high-quality SVBRDF acquisition: A new setup and method

Chuhua Xian; Jiaxin Li; Hao Wu; Zisen Lin; Guiqing Li

doi:10.1007/s41095-023-0352-6

| Sign up

PDF (20.8 MB)

Cite

EndNote(RIS) BibTeX

Collect

Submit Manuscript

Show Outline

Figures (23)

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Tables (9)

Table 1

Table 2

Table 3

Table 4

Table 5

Research Article | Open Access

Delving into high-quality SVBRDF acquisition: A new setup and method

Chuhua Xian^¹(), Jiaxin Li^¹, Hao Wu^², Zisen Lin^², Guiqing Li^¹

1School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China

2Guangdong Shidi Intelligence Technology, Ltd., Guangzhou 510000, China

Show Author Information

Graphical Abstract

View original image Download original image

Abstract

In this study, we present a new and innovative framework for acquiring high-quality SVBRDF maps. Our approach addresses the limi-tations of the current methods and proposes a new solution. The core of our method is a simple hardware setup consisting of a consumer-level camera, LED lights, and a carefully designed network that can accurately obtain the high-quality SVBRDF properties of a nearly planar object. By capturing a flexible number of images of an object, our network uses different subnetworks to train different property maps and employs appropriate loss functions for each of them. To further enhance the quality of the maps, we improved the network structure by adding a novel skip connection that connects the encoder and decoder withglobal features. Through extensive experimentation using both synthetic and real-world materials, our results demonstrate that our method outperforms previous methods and produces superior results. Furthermore, our proposed setup can also be used to acquire physically based rendering maps of special materials.

Keywords

acquisition setup SVBRDF acquisition material capture global skip connection

References

[1]

Holroyd,

; Lawrence,

; Zickler,

A coaxial optical scanner for synchronous acquisition of 3D geometry and surface reflectance. ACM Transactions on Graphics Vol. 29, No. 4, Article No. 99, 2010.

Crossref Google Scholar

[2]

Dana,

K. J.

; van Ginneken,

; Nayar,

S. K.

; Koenderink,

J. J.

Reflectance and texture of real-world surfaces. ACM Transactions on Graphics Vol. 18, No. 1, 1–34, 1999.

Crossref Google Scholar

[3]

Lawrence,

; Ben-Artzi,

; DeCoro,

; Matusik,

; Pfister,

; Ramamoorthi,

; Rusinkiewicz,

Inverse shade trees for non-parametric material representation and editing. ACM Transactions on Graphics Vol. 25, No. 3, 735–745, 2006.

Crossref Google Scholar

[4]

Li,

Z. Q.

; Sunkavalli,

; Chandraker,

Materials for masses: SVBRDF acquisition with a single mobile phone image. In: Computer Vision – ECCV 2018. Lecture Notes in Computer Science, Vol. 11207. Ferrari,

; Hebert,

; Sminchisescu,

; Weiss,

Eds. Springer Cham, 74–90, 2018.

Crossref

[5]

Deschaintre,

; Aittala,

; Durand,

; Drettakis,

; Bousseau,

Single-image SVBRDF capture with a rendering-aware deep network. ACM Transactions on Graphics Vol. 37, No. 4, Article No. 128, 2018.

Crossref Google Scholar

[6]

Deschaintre,

; Aittala,

; Durand,

; Drettakis,

; Bousseau,

Flexible SVBRDF capture with a multi-image deep network. Computer Graphics Forum Vol. 38, No. 4, 1–13, 2019.

Crossref Google Scholar

[7]

Asselin,

L. P.

; Laurendeau,

; Lalonde,

J. F.

Deep SVBRDF estimation on real materials. In: Proceedings of the International Conference on 3D Vision, 1157–1166, 2020.

Crossref

[8]

Guo,

; Lai,

S. C.

; Tao,

C. Z.

; Cai,

Y. L.

; Wang,

; Guo,

Y. W.

; Yan,

L. Q.

Highlight-aware two-stream network for single-image SVBRDF acquisition. ACM Transactions on Graphics Vol. 40, No. 4, Article No. 123, 2021.

Crossref Google Scholar

[9]

Zhao,

Y. Z.

; Wang,

B. B.

; Xu,

Y. N.

; Zeng,

; Wang,

; Holzschuch,

Joint SVBRDF recovery and synthesis from a single image using an unsupervised generative adversarial network. In: Proceedings of the Eurographics Symposium on Rendering, 53–66, 2020.

[10]

Wen,

; Wang,

B. B.

; Zhang,

; Guo,

; Holzschuch,

SVBRDF recovery from a single image with highlights using a pre-trained generative adversarial network. Computer Graphics Forum Vol. 41, No. 6, 110–123, 2022.

Crossref Google Scholar

[11]

Kang,

K. Z.

; Xie,

C. H.

; He,

C. G.

; Yi,

M. Q.

; Gu,

M. Y.

; Chen,

Z. M.

; Zhou,

; Wu,

H. Z.

Learning efficient illumination multiplexing for joint capture of reflectance and shape. ACM Transactions on Graphics Vol. 38, No. 6, Article No. 165, 2019.

Crossref Google Scholar

[12]

Zhou,

X. L.

; Kalantari,

N. K.

Adversarial single-image SVBRDF estimation with hybrid training. Computer Graphics Forum Vol. 40, No. 2, 315–325, 2021.

Crossref Google Scholar

[13]

Guo,

; Smith,

; Hasan,

; Sunkavalli,

; Zhao,

MaterialGAN: Reflectance capture using a generative SVBRDF model. ACM Transactions on Graphics Vol. 39, No. 6, Article No. 254, 2020.

Crossref Google Scholar

[14]

Gao,

; Li,

; Dong,

; Peers,

; Xu,

; Tong,

Deep inverse rendering for high-resolution SVBRDF estimation from an arbitrary number of images. ACM Transactions on Graphics Vol. 38, No. 4, Article No. 134, 2019.

Crossref Google Scholar

[15]

Ye,

W. J.

; Dong,

; Peers,

; Guo,

B. N.

Deep reflectance scanning: Recovering spatially-varying material appearance from a flash-lit video sequence. Computer Graphics Forum Vol. 40, No. 6, 409–427, 2021.

Crossref Google Scholar

[16]

Albert,

R. A.

; Chan,

D. Y.

; Goldman,

D. B.

; O’Brien,

J. F.

Approximate svBRDF estimation from mobile phone video. In: Proceedings of the Eurographics Symposium on Rendering: Experimental Ideas & Implementations, 11–22, 2018.

[17]

Baek,

S. H.

; Jeon,

D. S.

; Tong,

; Kim,

M. H.

Simultaneous acquisition of polarimetric SVBRDF and normals. ACM Transactions on Graphics Vol. 37, No. 6, Article No. 268, 2018.

Crossref Google Scholar

[18]

Deschaintre,

; Lin,

Y. M.

; Ghosh,

Deep polarization imaging for 3D shape and SVBRDF acquisition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15562–15571, 2021.

Crossref

[19]

Ma,

X. H.

; Kang,

K. Z.

; Zhu,

R. S.

; Wu,

H. Z.

; Zhou,

Free-form scanning of non-planar appearance with neural trace photography. ACM Transactions on Graphics Vol. 40, No. 4, Article No. 124, 2021.

Crossref Google Scholar

[20]

Tunwattanapong,

; Fyffe,

; Graham,

; Busch,

; Yu,

X. M.

; Ghosh,

; Debevec,

Acquiring reflectance and shape from continuous spherical harmonic illumination. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 109, 2013.

Crossref Google Scholar

[21]

Nam,

; Lee,

J. H.

; Gutierrez,

; Kim,

M. H.

Practical SVBRDF acquisition of 3D objects with unstructured flash photography. ACM Transactions on Graphics Vol. 37, No. 6, Article No. 267, 2018.

Crossref Google Scholar

[22]

Xia,

; Dong,

; Peers,

; Tong,

Recovering shape and spatially-varying surface reflectance under unknown illumination. ACM Transactions on Graphics Vol. 35, No. 6, Article No. 187, 2016.

Crossref Google Scholar

[23]

Ronneberger,

; Fischer,

; Brox,

U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Lecture Notes in Computer Science, Vol. 9351. Navab,

; Hornegger,

; Wells,

; Frangi,

Eds. Springer Cham, 234–241, 2015.

Crossref

[24]

Zhou,

Z. W.

; Rahman Siddiquee,

M. M.

; Tajbakhsh,

; Liang,

J. M.

UNet++: A nested U-net architecture for medical image segmentation. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. DLMIA ML-CDS 2018. Lecture Notes in Computer Science, Vol. 11045. Stoyanov,

, et al. Eds. Springer Cham, 3–11, 2018.

Crossref

[25]

Hu,

; Shen,

; Sun,

Squeeze-and-excitation networks. In: Proceedings of the IEEE/CVF Con-ference on Computer Vision and Pattern Recognition, 7132–7141, 2018.

Crossref

[26]

X-Rite: TAC7 appearance scanner. 2017. Available at https://www.xrite.com/categories/appearance/total-appearance-capture-ecosystem/tac7

[27]

Cycles: Open source production rendering. 2022. Available at https://www.cycles-renderer.org/

[28]

Kingma,

D. P.

; Ba,

Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.

Google Scholar

[29]

Smith,

A. R.

; Blinn,

J. F.

Blue screen matting. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, 259–268, 1996.

Crossref

Computational Visual Media

Volume 10 Issue 3,
June 2024

Pages 523-541

DOI: 10.1007/s41095-023-0352-6

Cite this article:

Xian C, Li J, Wu H, et al. Delving into high-quality SVBRDF acquisition: A new setup and method. Computational Visual Media, 2024, 10(3): 523-541. https://doi.org/10.1007/s41095-023-0352-6

Return

10.1007/s41095-023-0352-6.T001Table 1RMSE comparisons between glossiness map generated w/ and w/o global skip connections (GloSkip)

	w/o GloSkip	w/ GloSkip
RMSE	0.052626	0.040709

10.1007/s41095-023-0352-6.T002Table 2RMSE comparisons on our dataset using two images as input. Here, $d$ , $n$ , $s$ , $g$ , and $r$ indicate the diffuse, normal, specular, glossiness, and the rendered image, respectively

	Deschaintre et al.	Guo J et al. (materialGAN)	Guo Y et al.	Ours
$d$	0.082861	0.006006	0.054556	Ours	0.000306
$n$	0.004437	0.005079	0.005232	0.000791
$s$	0.007402	0.013387	0.010090	0.046552
$g$	0.088811	0.132340	0.095743	0.044373
$r$	0.077482	0.009083	0.044927	0.000602

10.1007/s41095-023-0352-6.T003Table 3RMSE comparison between previous works and our method on real materials with 2 images input. The first row shows the metrics on re-renderings under $24$ lights using our device, while the second row is under novel light

	Deschaintre et al.	Guo J et al. (materialGAN)	Guo Y et al.	Ours
$24$	0.066352	0.017784	0.068793	Ours	0.013180
Novel	0.196887	0.087135	0.202598	0.053856

10.1007/s41095-023-0352-6.T004Table 4Image numbers we input in the experiment

Input	No.
$1$	$0$
$2$	$0, 16$
$3$	$0, 16, 23$
$6$	$0, 4, 8, 12, 16, 20$
$10$	$4, 6, 8, 10, 12, 14, 16, 18, 20, 22$
$16$	$0, 2, 4, 5, 6, 8, 10, 11, 12, 14, 16, 17, 18, 20, 22, 23$

10.1007/s41095-023-0352-6.T005Table 5RMSE comparisons on our dataset. $d$ , $n$ , $s$ , $g$ , and $r$ indicate the diffuse, normal, specular, glossiness, and the rendering image, respectively

	Deschaintre et al.	Guo J et al. (materialGAN)	Guo Y et al.	Ours
$d$	0.102023	0.005124	0.054556	Ours	0.000423
$n$	0.006479	0.004360	0.005232	0.000247
$s$	0.033632	0.012154	0.010090	0.025878
$g$	0.102216	0.140183	0.095743	0.040709
$r$	0.087551	0.006833	0.044927	0.000301

10.1007/s41095-023-0352-6.T006Table 6LPIPS comparisons on our dataset. $d$ , $n$ , $s$ , $g$ , and $r$ indicate the diffuse, normal, specular, glossiness, and rendering image, respectively

	Deschaintre et al.	Guo J et al. (materialGAN)	Guo Y et al.	Ours
$d$	0.306619	0.133610	0.286742	Ours	0.063140
$n$	0.205903	0.246912	0.364797	0.151345
$s$	0.716021	0.692563	0.747934	0.721130
$g$	0.656656	0.579437	0.519371	0.495239
$r$	0.299803	0.179935	0.318218	0.130414

10.1007/s41095-023-0352-6.T007Table 7RMSE comparisons between previous works and our method on real materials. The first row shows the metrics on re-renderings under $24$ lights using our device, while the second row shows the metrics under novel lights

	Deschaintre et al.	Guo J et al. (materialGAN)	Guo Y et al.	Ours
$24$	0.069047	0.013157	0.068793	Ours	0.012600
Novel	0.204769	0.072937	0.202598	0.067823

10.1007/s41095-023-0352-6.T008Table 8LPIPS comparisons between previous works and our method on real materials. The first row shows the metrics on re-renderings under $24$ lights using our device, while the second row shows the metrics under novel lights

	Deschaintre et al.	Guo J et al. (materialGAN)	Guo Y et al.	Ours
$24$	0.470221	0.377422	0.498515	Ours	0.284123
Novel	0.630545	0.516387	0.611689	0.422746

10.1007/s41095-023-0352-6.T009Table 9RMSE comparisons of the ablation study

	$1$ encoder	Global track	Ours
$d$	0.017557	0.021936	0.011774
$s$	0.019679	0.022555	0.018615
$n$	0.023288	0.036720	0.020182
$g$	0.037449	0.052994	0.032916