| Sign up

PDF (13.7 MB)

Cite

EndNote(RIS) BibTeX

Collect

Collect

Submit Manuscript

Show Outline

Figures (13)

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Tables (4)

Table 1

Table 2

Table 3

Table 4

Research Article | Open Access

Adaptive sampling and reconstruction for gradient-domain rendering

Yuzhi Liang^¹, Tao Liu^², Yuchi Huo^¹, Rui Wang^¹, Hujun Bao^¹()

1

State Key Lab of CAD&CG, Zhejiang University, Hangzhou 310058, China

2

College of Transport and Communications, Shanghai Maritime University, Shanghai 201306, China

Show Author Information

Graphical Abstract

View original image Download original image

Abstract

Gradient-domain rendering estimates finite difference gradients of image intensities and reconstructs the final result by solving a screened Poisson problem, which shows improvements over merely sampling pixel intensities. Adaptive sampling is another orthogonal research area that focuses on distributing samples adaptively in the primal domain. However, adaptive sampling in the gradient domain with low sampling budget has been less explored. Our idea is based on the observation that signals in the gradient domain are sparse, which provides more flexibility for adaptive sampling. We propose a deep-learning-based end-to-end sampling and reconstruction framework in gradient-domain rendering, enabling adaptive sampling gradient and the primal maps simultaneously. We conducted extensive experiments for evaluation and showed that our method produces better reconstruction quality than other methods in the test dataset.

Keywords

gradient-domain rendering adaptive rendering Monte Carlo rendering deep-learning-based Monte Carlo denoising

Electronic Supplementary Material

Download File(s)

cvm-10-5-885_ESM.zip (159.6 MB)

References

[1]

Kajiya, J. T. The rendering equation. ACM SIGGRAPH Computer Graphics Vol. 20, No. 4, 143–150, 1986.

Crossref Google Scholar

[2]

Zwicker, M.; Jarosz, W.; Lehtinen, J.; Moon, B.; Ramamoorthi, R.; Rousselle, F.; Sen, P.; Soler, C.; Yoon, S. E. Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. Computer Graphics Forum Vol. 34, No. 2, 667–681, 2015.

Crossref Google Scholar

[3]

Kuznetsov, A.; Kalantari, N. K.; Ramamoorthi, R. Deep adaptive sampling for low sample count rendering. Computer Graphics Forum Vol. 37, No. 4, 35–44, 2018.

Crossref Google Scholar

[4]

Hua, B. S.; Gruson, A.; Petitjean, V.; Zwicker, M.; Nowrouzezahrai, D.; Eisemann, E.; Hachisuka, T. A survey on gradient-domain rendering. Computer Graphics Forum Vol. 38, No. 2, 455–472, 2019.

Crossref Google Scholar

[5]

Lehtinen, J.; Karras, T.; Laine, S.; Aittala, M.; Durand, F.; Aila, T. Gradient-domain metropolis light transport. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 95, 2013.

Crossref Google Scholar

[6]

Kettunen, M.; Manzi, M.; Aittala, M.; Lehtinen, J.; Durand, F.; Zwicker, M. Gradient-domain path tracing. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 123, 2015.

Crossref Google Scholar

[7]

Hua, B. S.; Gruson, A.; Nowrouzezahrai, D.; Hachisuka, T. Gradient-domain photon density estimation. Computer Graphics Forum Vol. 36, No. 2, 31–38, 2017.

Crossref Google Scholar

[8]

Manzi, M.; Kettunen, M.; Aittala, M.; Lehtinen, J.; Durand, F.; Zwicker, M. Gradient-domain bidirectional path tracing. In: Proceedings of the Eurographics Symposium on Rendering: Experimental Ideas & Implementations, 65–74, 2015.

[9]

Sun, W.; Sun, X.; Carr, N.; Nowrouzezahrai, D.; Ramamoorthi, R. Gradient-domain vertex connection and merging. In: Proceedings of the Eurographics Symposium on Rendering: Experimental Ideas & Implementations, 83–92, 2017.

[10]

Manzi, M.; Kettunen, M.; Durand, F.; Zwicker, M.; Lehtinen, J. Temporal gradient-domain path tracing. ACM Transactions on Graphics Vol. 35, No. 6, Article No. 246, 2016.

Crossref Google Scholar

[11]

Huo, Y.; Yoon, S. E. A survey on deep learning-based Monte Carlo denoising. Computational Visual Media Vol. 7, No. 2, 169–185, 2021.

Crossref Google Scholar

[12]

Kettunen, M.; Härkönen, E.; Lehtinen, J. Deep convolutional reconstruction for gradient-domain rendering. ACM Transactions on Graphics Vol. 38, No. 4, Article No. 126, 2019.

Crossref Google Scholar

[13]

Guo, J.; Li, M.; Li, Q.; Qiang, Y.; Hu, B.; Guo, Y.; Yan, L. Q. GradNet: Unsupervised deep screened Poisson reconstruction for gradient-domain rendering. ACM Transactions on Graphics Vol. 38, No. 6, Article No. 223, 2019.

Crossref Google Scholar

[14]

Hasselgren, J.; Munkberg, J.; Salvi, M.; Patney, A.; Lefohn, A. Neural temporal adaptive sampling and denoising. Computer Graphics Forum Vol. 39, No. 2, 147–155, 2020.

Crossref Google Scholar

[15]

Veach, E. Robust Monte Carlo methods for light transport simulation. Ph.D. Dissertation. Stanford University, 1997.

[16]

Hachisuka, T.; Jensen, H. W. Stochastic progressive photon mapping. ACM Transactions on Graphics Vol. 28, No. 5, Article No. 141, 2009.

Crossref Google Scholar

[17]

Georgiev, I. Implementing vertex connection and merging. Technical Report. Saarland University, 2012. Available at https://iliyan.com/publications/ImplementingVCM/ImplementingVCM_TechRep2012_rev2.pdf

[18]

Bauszat, P.; Petitjean, V.; Eisemann, E. Gradient-domain path reusing. ACM Transactions on Graphics Vol. 36, No. 6, Article No. 229, 2017.

Crossref Google Scholar

[19]

Gruson, A.; Hua, B. S.; Vibert, N.; Nowrouzezahrai, D.; Hachisuka, T. Gradient-domain volumetric photon density estimation. ACM Transactions on Graphics Vol. 37, No. 4, Article No. 82, 2018.

Crossref Google Scholar

[20]

Jarosz, W.; Zwicker, M.; Jensen, H. W. The beam radiance estimate for volumetric photon mapping. Computer Graphics Forum Vol. 27, No. 2, 557–566, 2008.

Crossref Google Scholar

[21]

Jarosz, W.; Nowrouzezahrai, D.; Thomas, R.; Sloan, P. P.; Zwicker, M. Progressive photon beams. ACM Transactions on Graphics Vol. 30, No. 5, 1–12, 2011.

Crossref Google Scholar

[22]

Bitterli, B.; Jarosz, W. Beyond points and beams: Higher-dimensional photon samples for volumetric light transport. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 112, 2017.

Crossref Google Scholar

[23]

Petitjean, V.; Bauszat, P.; Eisemann, E. Spectral gradient sampling for path tracing. Computer Graphics Forum Vol. 37, No. 4, 45–53, 2018.

Crossref Google Scholar

[24]

Back, J.; Yoon, S. E.; Moon, B. Feature generation for adaptive gradient-domain path tracing. Computer Graphics Forum Vol. 37, No. 7, 65–74, 2018.

Crossref Google Scholar

[25]

Moon, B.; McDonagh, S.; Mitchell, K.; Gross, M. Adaptive polynomial rendering. ACM Transactions on Graphic Vol. 35, No. 4, Article No. 40, 2016.

Crossref Google Scholar

[26]

Chaitanya, C. R. A.; Kaplanyan, A. S.; Schied, C.; Salvi, M.; Lefohn, A.; Nowrouzezahrai, D.; Aila, T. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 98, 2017.

Crossref Google Scholar

[27]

Bako, S.; Vogels, T.; McWilliams, B.; Meyer, M.; NováK, J.; Harvill, A.; Sen, P.; Derose, T.; Rousselle, F. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 97, 2017.

Crossref Google Scholar

[28]

Vogels, T.; Rousselle, F.; McWilliams, B.; Röthlin, G.; Harvill, A.; Adler, D.; Meyer, M.; Novák, J. Denoising with kernel prediction and asymmetric loss functions. ACM Transactions on Graphics Vol. 37, No. 4, Article No. 124, 2018.

Crossref Google Scholar

[29]

Xu, B.; Zhang, J.; Wang, R.; Xu, K.; Yang, Y. L.; Li, C.; Tang, R. Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation. ACM Transactions on Graphics Vol. 38, No. 6, Article No. 224, 2019.

Crossref Google Scholar

[30]

Goodfellow, I. J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y.; Haloui, I.; Gupta, J. S.; et al. Generative adversarial networks. arXiv preprint arXiv:1406.2661, 2014.

[31]

Lin, W.; Wang, B.; Wang, L.; Holzschuch, N. A detail preserving neural network model for Monte Carlo denoising. Computational Visual Media Vol. 6, No. 2, 157–168, 2020.

Crossref Google Scholar

[32]

Lin, W.; Wang, B.; Yang, J.; Wang, L.; Yan, L. Q. Path-based Monte Carlo denoising using a three-scale neural network. Computer Graphics Forum Vol. 40, No. 1, 369–381, 2021.

Crossref Google Scholar

[33]

Kettunen, M.; Härkönen, E.; Lehtinen, J. Deep convolutional reconstruction for gradient-domain rendering. ACM Transactions on Graphics Vol. 38, No. 4, Article No. 126, 2019.

Crossref Google Scholar

[34]

Huang, G.; Liu, Z.; van der Maaten, L.; Weinberger, K. Q. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016.

[35]

Xu, Z.; Sun, Q.; Wang, L.; Xu, Y.; Wang, B. Unsupervised image reconstruction for gradient-domain volumetric rendering. Computer Graphics Forum Vol. 39, No. 7, 193–203, 2020.

Crossref Google Scholar

[36]

Clevert, D. A.; Unterthiner, T.; Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). arXiv preprint arXiv:1511.07289, 2015.

[37]

Kettunen, M.; Härkönen, E.; Lehtinen, J. E-LPIPS: Robust perceptual image similarity via random transformation ensembles. arXiv preprint arXiv:1906. 03973, 2019.

[38]

Zhang, R.; Isola, P.; Efros, A. A.; Shechtman, E.; Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 586–595, 2018.

[39]

Reinhard, E.; Stark, M.; Shirley, P.; Ferwerda, J. Photographic tone reproduction for digital images. ACM Transactions on Graphics Vol. 21, No. 3, 267–276, 2002.

Crossref Google Scholar

[40]

Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. PyTorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019.

[41]

Bitterli, B. Rendering resources. 2016. Available at https://benedikt-bitterli.me/resources/

[42]

Manzi, M.; Rousselle, F.; Kettunen, M.; Lehtinen, J.; Zwicker, M. Improved sampling for gradient-domain metropolis light transport. ACM Transactions on Graphics Vol. 33, No. 6, Article No. 178, 2014.

Crossref Google Scholar

[43]

Chaitanya, C. R. A.; Kaplanyan, A. S.; Schied, C.; Salvi, M.; Lefohn, A.; Nowrouzezahrai, D.; Aila, T. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 98, 2017.

Crossref Google Scholar

[44]

Huang, G.; Liu, Z.; Van Der Maaten, L.; Weinberger, K. Q. Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2261–2269, 2017.

Computational Visual Media

Volume 10 Issue 5,
October 2024

Pages 885-902

DOI: 10.1007/s41095-023-0361-5

Cite this article:

Liang Y, Liu T, Huo Y, et al. Adaptive sampling and reconstruction for gradient-domain rendering. Computational Visual Media, 2024, 10(5): 885-902. https://doi.org/10.1007/s41095-023-0361-5

Fig. 1 Illustration of forward and backward shift mapping between pixel $i$ and pixel $j$ . After applying shift mapping, primal paths in space $S_{i}$ are mapped to space $T (S_{i})$ . Owing to visibility issues, shift mapping is not always successful, the overlapped shaded areas are spaces where both forward and backward mapping is valid.
Fig. 2 Our method consists of three sub-modules, as shown in the rounded rectangles. The sampling map estimator is fed with uniform 1 spp buffers and predicts sampling maps to guide the gradient-domain renderer. The rendering of $I_{d x}$ is guided by right and left sampling maps, whereas the rendering of $I_{d y}$ is guided by top and bottom sampling maps. The rendering of $I_{p}$ and auxiliary features are guided by primal sampling map. Finally, the denoiser reconstructs denoised result from adaptive rendered images.
Fig. 3 Fitted curves of gradient contributions of two pixels. The curve “original” is the densely sampled original signal, whereas the curve “fitted” is the fitted polynomial function.
Fig. 4 This figure displays sampling maps and our rendered images of two selected scenes. For each scene, the first row shows the rendered results of our full method, the second row shows the corresponding sampling maps, and the third row shows the sampling maps of heuristic methods.
Fig. 5 Comparison of adaptive sampling only in the primal domain. We keep the weighted sum of the sampling budget to be the same, when comparing different methods.
Fig. 6 Comparison with uniform sampling on “dining room” scene and “kitchen” scene. We retain the weighted sum of sampling budget to be the same when comparing between different methods.
Fig. 7 Comparison with uniform sampling on “spaceship” scene and “house” scene. We retain the weighted sum of sampling budget to be the same when comparing between different methods.
Fig. 8 Additional results of comparison. “TGPT” denotes the first heuristic method used for comparison; it calculates sampling map by combining variances of $I_{d x}$ , $I_{d y}$ , and $I_{p}$ . “ $V a r_I n t e n s i t y$ ” denotes the second heuristic method used for comparison.
Fig. 9 Failure reconstruction on bedroom scene.
Fig. 10 UNet-based sampling map estimator.
Fig. 11 Dense-UNet-based denoiser.
Fig. 12 Sampling map comparison between our full method and $V a r_I n t e n s i t y$ .
Fig. 13 Comparison of sampling maps using our method with single sampling map, TGPT, and DASR.

Return

Table 1 Error metrics of different methods. All the metrics are scaled by 1000

Scene		Ours	w.o. gradients	w.o. adaptive
Dining room	RMSE	7.17	7.50	11.98
Dining room	E-LPIPS	2.48	3.74	3.66
Spaceship	RMSE	2.95	3.04	7.05
Spaceship	E-LPIPS	1.27	1.56	2.13
House	RMSE	9.30	18.3	11.95
House	E-LPIPS	2.00	3.50	2.65
Kitchen	RMSE	3.55	4.16	6.32
Kitchen	E-LPIPS	1.33	1.55	1.57
Sponza	RMSE	3.82	4.23	4.44
Sponza	E-LPIPS	1.93	3.16	2.46

Table 2 Samples allocation for test scenes of different methods

Scene		Ours	w.o. gradients	w.o. adaptive
Dining room	$I_{p}$	7.2	10	4
	$I_{d x}$	1.8	0	4
	$I_{d y}$	2.1	0	4
Spaceship	$I_{p}$	7.5	10	4
	$I_{d x}$	1.5	0	4
	$I_{d y}$	1.8	0	4
House	$I_{p}$	6.2	10	4
	$I_{d x}$	2.6	0	4
	$I_{d y}$	2.5	0	4
Kitchen	$I_{p}$	7.1	10	4
	$I_{d x}$	1.8	0	4
	$I_{d y}$	2.1	0	4
Sponza	$I_{p}$	6.7	10	4
	$I_{d x}$	2.1	0	4
	$I_{d y}$	2.3	0	4

Table 3 Equal time comparison with NGPT

Scene	Ours RMSE	NGPT RMSE
dining room	0.0072 (4 spp/14.5 s)	0.012 (4 spp/14.8 s)
spaceship	0.00295 (4 spp/11.9 s)	0.0070 (4 spp/12.1 s)
house	0.0093 (4 spp/12.8 s)	0.012 (6 spp/12.6 s)
kitchen	0.0035 (4 spp/14.5 s)	0.0063 (4 spp/14.9 s)
sponza	0.00382 (4 spp/14.9 s)	0.0044 (4 spp/15.3 s)

Table 4 GPU inference time

Module	Time (ms)
Sampling map estimator	25
Denoiser	195

About Us

Learn about Open Access

Tsinghua University Press

Publish with Us

Peer Review Policy

Copyright and Licensing

Article Processing Charge

Contact Us

Journal Collaboration: Yao Meng (Ms.)✉️ +86-10-83470574

Technical Support: Kuo Zhao (Mr.)✉️ +86-10-83470507

Media Contact: Hao Jin (Mr.)✉️ +86-10-83470559

Address: Floor 6, Tower B, Xueyan Building, Shuangqing Road, Haidian District, Beijing 100084, China.

SciOpen——中国科技期刊卓越行动计划支持项目

Copyright © 2025 Tsinghua University Press Ltd.

京ICP备 10035462号-42 京公网安备11010802044758号