AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Computational Visual Media Article
PDF (9.9 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Review Article | Open Access

A survey on deep learning-based Monte Carlo denoising

KAIST, Daejeon, 31414, Republic of Korea
Show Author Information

Abstract

Monte Carlo (MC) integration is used ubiquitously in realistic image synthesis because of its flexibility and generality. However, the integration has to balance estimator bias and variance, which causes visually distracting noise with low sample counts. Existing solutions fall into two categories, in-process sampling schemes and post-processing reconstruction schemes. This report summarizes recent trends in the post-processing reconstruction scheme. Recent years have seen increasing attention and significant progress in denoising MC rendering with deep learning, by training neural networks to reconstruct denoised rendering results from sparse MC samples. Many of these techniques show promising results in real-world applications, and this report aims to provide an assessment of these approaches for practitioners and researchers.

Keywords

renderingMonte Carlo (MC) denoisingdeep learningray tracing

References

[1]
Kajiya, J. T. The rendering equation. In: Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques, 143-150, 1986.
Crossref
[2]
Pharr, M.; Jakob, W.; Humphreys, G. Physically based Rendering: From Theory to Implementation. Morgan Kaufmann, 2016.
[3]
Rubinstein, R. Y.; Kroese, D. P. Simulation and the Monte Carlo Method. Hoboken, NJ, USA: John Wiley & Sons, Inc., 2016.
Crossref
[4]
LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature Vol. 521, No. 7553, 436-444, 2015.
Crossref Google Scholar
[5]
Goodfellow, I.; Bengio, Y.; Courville, A.; Bengio, Y. Deep Learning. MIT Press, 2016.
[6]
Zwicker, M.; Jarosz, W.; Lehtinen, J.; Moon, B.; Ramamoorthi, R.; Rousselle, F.; Sen, P.; Soler, C.; Yoon, S.-E. Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. Computer Graphics Forum Vol. 34, No. 2, 667-681, 2015.
Crossref Google Scholar
[7]
Dahlberg, H.; Adler, D.; Newlin, J. Machine-learning denoising in feature film production. In: Proceedings of the ACM SIGGRAPH 2019 Talks, Article No. 21, 2019.
Crossref
[8]
Chaitanya, C. R. A.; Kaplanyan, A. S.; Schied, C.; Salvi, M.; Lefohn, A.; Nowrouzezahrai, D.; Aila, T. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 98, 2017.
Crossref Google Scholar
[9]
Vogels, T.; Rousselle, F.; McWilliams, B.; Röthlin, G.; Harvill, A.; Adler, D.; Meyer, M.; Novák, J. Denoising with kernel prediction and asymmetric loss functions. ACM Transactions on Graphics Vol. 37, No. 4, Article No. 124, 2018.
Crossref Google Scholar
[10]
Kalantari, N. K.; Bako, S.; Sen, P. A machine learning approach for filtering Monte Carlo noise. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 122, 2015.
Crossref Google Scholar
[11]
Hastie, T.; Tibshirani, R.; Friedman, J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. Springer Science & Business Media, 2009.
[12]
Rosenblatt, F. Principles of neurodynamics. perceptrons and the theory of brain mechanisms. Technical Report. Cornell Aeronautical Lab Inc Buffalo NY, 1961.
Crossref
[13]
Rumelhart, D. E.; Hinton, G. E.; Williams, R. J. Learning internal representations by error propagation. Technical Report. California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Crossref
[14]
Xing, Q. W.; Chen, C. Y. Path tracing denoising based on SURE adaptive sampling and neural network. IEEE Access Vol. 8, 116336-116349, 2020.
Crossref Google Scholar
[15]
Stein, C. M. Estimation of the mean of a multivariate normal distribution. The Annals of Statistics Vol. 9, No. 6, 1135-1151, 1981.
Crossref Google Scholar
[16]
Bako, S.; Vogels, T.; McWilliams, B.; Meyer, M.; NováK, J.; Harvill, A.; Sen, P.; Derose, T.; Rousselle, F. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 97, 2017.
Crossref Google Scholar
[17]
Gharbi, M.; Li, T.-M.; Aittala, M.; Lehtinen, J.; Durand, F. Sample-based Monte Carlo denoising using a kernel-splatting network. ACM Transactions on Graphics Vol. 38, No. 4, Article No. 125, 2019.
Crossref Google Scholar
[18]
LeCun, Y.; Boser, B.; Denker, J. S.; Henderson, D.; Howard, R. E.; Hubbard, W.; Jackel, L. D. Backpropagation applied to handwritten zip code recognition. Neural Computation Vol. 1, No. 4, 541-551, 1989.
Crossref Google Scholar
[19]
LeCun, Y.; Boser, B. E.; Denker, J. S.; Henderson, D.; Howard, R. E.; Hubbard, W. E.; Jackel, L. D. Handwritten digit recognition with a back-propagation network. In: Proceedings of the 2nd International Conference on Neural Information Processing Systems, 396-404, 1989.
[20]
Back, J.; Hua, B.-S.; Hachisuka, T.; Moon, B. Deep combiner for independent and correlated pixel estimates. ACM Transactions on Graphics Vol. 39, No. 6, Article No. 242, 2020.
Crossref Google Scholar
[21]
Xu, B.; Zhang, J. F.; Wang, R.; Xu, K.; Yang, Y. L.; Li, C.; Tang, R. Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation. ACM Transactions on Graphics Vol. 38, No. 6, Article No. 224, 2019.
Crossref Google Scholar
[22]
Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, Vol. 2, 2672-2680, 2014.
[23]
Creswell, A.; White, T.; Dumoulin, V.; Arulkumaran, K.; Sengupta, B.; Bharath, A. A. Generative adversarial networks: An overview. IEEE Signal Processing Magazine Vol. 35, No. 1, 53-65, 2018.
Crossref Google Scholar
[24]
Alsaiari, A.; Rustagi, R.; Thomas, M. M.; Forbes, A. G. Image denoising using a generative adversarial network. In: Proceedings of the IEEE 2nd International Conference on Information and Computer Technologies, 126-132, 2019.
Crossref
[25]
He, K. M.; Zhang, X. Y.; Ren, S. Q.; Sun, J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770-778, 2016.
[26]
Wong, K. M.; Wong, T. T. Deep residual learning for denoising Monte Carlo renderings. Computational Visual Media Vol. 5, No. 3, 239-255, 2019.
Crossref Google Scholar
[27]
Yang, X.; Wang, D. W.; Hu, W. B.; Zhao, L. J.; Piao, X. L.; Zhou, D. S.; Zhang, Q.; Yin, B.; Cai, Q.; Wei, X. Fast reconstruction for Monte Carlo rendering using deep convolutional networks. IEEE Access Vol. 7, 21177-21187, 2019.
Crossref Google Scholar
[28]
Yang, X.; Wang, D. W.; Hu, W. B.; Zhao, L. J.; Yin, B. C.; Zhang, Q.; Wei, X.-P.; Fu, H. DEMC: A deep dual-encoder network for denoising Monte Carlo rendering. Journal of Computer Science and Technology Vol. 34, No. 5, 1123-1135, 2019.
Crossref Google Scholar
[29]
Ballard, D. H. Modular learning in neural networks. In: Proceedings of the 6th National Conference on Artificial Intelligence, Vol. 1, 279-284, 1987.
[30]
Vincent, P.; Larochelle, H.; Bengio, Y.; Manzagol, P. A. Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, 1096-1103, 2008.
Crossref
[31]
Kuznetsov, A.; Kalantari, N. K.; Ramamoorthi, R. Deep adaptive sampling for low sample count rendering. Computer Graphics Forum Vol. 37, No. 4, 35-44, 2018.
Crossref Google Scholar
[32]
Munkberg, J.; Hasselgren, J. Neural denoising with layer embeddings. Computer Graphics Forum Vol. 39, No. 4, 1-12, 2020.
Crossref Google Scholar
[33]
Hanika, J.; Droske, M.; Fascione, L. Manifold next event estimation. Computer Graphics Forum Vol. 34, No. 4, 87-97, 2015.
Crossref Google Scholar
[34]
Lin, W. H.; Wang, B. B.; Yang, J.; Wang, L.; Yan, L. Q. Path-based Monte Carlo denoising using a three-scale neural network. Computer Graphics Forum Vol. 40, 369-381, 2021.
Crossref Google Scholar
[35]
Levoy, M.; Hanrahan, P. Light field rendering. In:Proceedings of the 23rd Annual Conference on ComputerGraphics and Interactive Techniques, 31-42, 1996.
Crossref
[36]
Lin, W. H.; Wang, B. B.; Wang, L.; Holzschuch, N. A detail preserving neural network model for Monte Carlo denoising. Computational Visual Media Vol. 6, No. 2, 157-168, 2020.
Crossref Google Scholar
[37]
Durand, F.; Holzschuch, N.; Soler, C.; Chan, E.; Sillion, F. X. A frequency analysis of light transport. ACM Transactions on Graphics Vol. 24, No. 3, 1115-1126, 2005.
Crossref Google Scholar
[38]
Belcour, L.; Soler, C.; Subr, K.; Holzschuch, N.; Durand, F. 5D Covariance tracing for efficient defocus and motion blur. ACM Transactions on Graphics Vol. 32, No. 3, Article No. 31, 2013.
Crossref Google Scholar
[39]
Liang, Y. L.; Wang, B. B.; Wang, L.; Holzschuch, N. Fast computation of single scattering in participating media with refractive boundaries using frequency analysis. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 10, 2961-2969, 2020.
Crossref Google Scholar
[40]
Bako, S.; Meyer, M.; DeRose, T.; Sen, P. Offline deep importance sampling for Monte Carlo path tracing.Computer Graphics Forum Vol. 38, No. 7, 527-542, 2019.
Crossref Google Scholar
[41]
Huo, Y.; Wang, R.; Zheng, R.; Xu, H.; Bao, H.; Yoon, S.-E. Adaptive incident radiance field sampling and reconstruction using deep reinforcement learning. ACM Transactions on Graphics Vol. 39, No. 1, Article No. 6, 2020.
Crossref Google Scholar
[42]
Jiang, G.; Kainz, B. Deep radiance caching: Convolutional autoencoders deeper in ray tracing. Computers & Graphics Vol. 94, 22-31, 2021.
Crossref Google Scholar
[43]
Lehtinen, J.; Karras, T.; Laine, S.; Aittala, M.; Durand, F.; Aila, T. Gradient-domain metropolis light transport. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 95, 2013.
Crossref Google Scholar
[44]
Kettunen, M.; Manzi, M.; Aittala, M.; Lehtinen, J.; Durand, F.; Zwicker, M. Gradient-domain path tracing. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 123, 2015.
Crossref Google Scholar
[45]
Hua, B. S.; Gruson, A.; Petitjean, V.; Zwicker, M.; Nowrouzezahrai, D.; Eisemann, E.; Hachisuka, T. A survey on gradient-domain rendering. Computer Graphics Forum Vol. 38, No. 2, 455-472, 2019.
Crossref Google Scholar
[46]
Kettunen, M.; Härkönen, E.; Lehtinen, J. Deep convolutional reconstruction for gradient-domain rendering. ACM Transactions on Graphics Vol. 38, No. 4, Article No. 126, 2019.
Crossref Google Scholar
[47]
Guo, J.; Li, M.; Li, Q.; Qiang, Y.; Hu, B.; Guo, Y.; Yan, L.-Q. GradNet: Unsupervised deep screened poisson reconstruction for gradient-domain rendering. ACM Transactions on Graphics Vol. 38, No. 6, Article No. 223, 2019.
Crossref Google Scholar
[48]
Jensen, H. W. Realistic Image Synthesis Using Photon Mapping. AK Peters/CRC Press, 2001.
Crossref
[49]
Kang, C. M.; Wang, L.; Xu, Y. N.; Meng, X. X. A survey of photon mapping state-of-the-art research and future challenges. Frontiers of Information Technology & Electronic Engineering Vol. 17, No. 3, 185-199, 2016.
Crossref Google Scholar
[50]
Zhu, S.; Xu, Z.; Jensen, H. W.; Su, H.; Ramamoorthi, R. Deep kernel density estimation for photon mapping. Computer Graphics Forum Vol. 39, No. 4, 35-45, 2020.
Crossref Google Scholar
[51]
Hachisuka, T.; Ogaki, S.; Jensen, H. W. Progressive photon mapping. In: Proceedings of the ACM SIGGRAPH Asia 2008 papers, Article No. 130, 2008.
Crossref
[52]
Zeng, Z.; Wang, L.; Wang, B. B.; Kang, C. M.; Xu, Y. N. Denoising stochastic progressive photon mapping renderings using a multi-residual network. Journal of Computer Science and Technology Vol. 35, No. 3, 506-521, 2020.
Crossref Google Scholar
[53]
Rumelhart, D. E.; Hinton, G. E.; Williams, R. J. Learning representations by back-propagating errors. Nature Vol. 323, No. 6088, 533-536, 1986.
Crossref Google Scholar
[54]
Huang, Y.; Wang, W.; Wang, L. Bidirectional recurrent convolutional networks for multi-frame super-resolution. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, Vol. 1, 235-243, 2015.
[55]
Mehta, S. U.; Wang, B.; Ramamoorthi, R. Axis-aligned filtering for interactive sampled soft shadows. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 163, 2012.
Crossref Google Scholar
[56]
Dammertz, H.; Sewtz, D.; Hanika, J.; Lensch, H. P. A. Edge-avoiding À-Trous wavelet transform for fast global illumination filtering. In: Proceedings of the Conference on High Performance Graphics, 67-75, 2010.
[57]
Li, T. M.; Wu, Y. T.; Chuang, Y. Y. SURE-based optimization for adaptive sampling and reconstruction. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 194, 2012.
Crossref Google Scholar
[58]
Hasselgren, J.; Munkberg, J.; Salvi, M.; Patney, A.; Lefohn, A. Neural temporal adaptive sampling and denoising. Computer Graphics Forum Vol. 39, No. 2, 147-155, 2020.
Crossref Google Scholar
[59]
Meng, X.; Zheng, Q.; Varshney, A.; Singh, G.; Zwicker, M. Real-time Monte Carlo denoising with the neural bilateral grid. In: Proceedings of the Eurographics Symposium on Rendering, 2020.
[60]
Drebin, R. A.; Carpenter, L.; Hanrahan, P. Volume rendering. ACM SIGGRAPH Computer Graphics Vol. 22, No. 4, 65-74, 1988.
Crossref Google Scholar
[61]
Max, N. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics Vol. 1, No. 2, 99-108, 1995.
Crossref Google Scholar
[62]
Kallweit, S.; Müller, T.; McWilliams, B.; Gross, M.; Novák, J. Deep scattering: Rendering atmospheric clouds with radiance-predicting neural networks. ACM Transactions on Graphics Vol. 36, No. 6, Article No. 231, 2017.
Crossref Google Scholar
[63]
Panin, M.; Nikolenko, S. Faster RPNN: Rendering clouds with latent space light probes. In: Proceedings of the SIGGRAPH Asia 2019 Technical Briefs, 21-24, 2019.
Crossref
[64]
Xu, Z. L.; Sun, Q.; Wang, L.; Xu, Y. N.; Wang, B. B. Unsupervised image reconstruction for gradient-domain volumetric rendering. Computer Graphics Forum Vol. 39, No. 7, 193-203, 2020.
Crossref Google Scholar
[65]
Hofmann, N.; Martschinke, J.; Engel, K.; Stamminger, M. Neural denoising for path tracing of medical volumetric data. In: Proceedings of the ACM on Computer Graphics and Interactive Techniques, Article No. 13, 2020.
Crossref
[66]
Jensen, H. W. Importance driven path tracing using the photon map. In: Rendering Techniques ’95. Hanrahan, P. M.; Purgathofer, W. Eds. Springer Vienna, 326-335, 1995.
Crossref
[67]
Hey, H.; Purgathofer, W. Importance sampling with hemispherical particle footprints. In: Proceedings of the 18th Spring Conference on Computer Graphics, 107-114, 2002.
Crossref
[68]
Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G. et al. Human-level control through deep reinforcement learning. Nature Vol. 518, No. 7540, 529-533, 2015.
Crossref Google Scholar
[69]
Silver, D.; Huang, A.; Maddison, C. J.; Guez, A.; Sifre, L.; van den Driessche, G.; Schrittwieser, J.; Antonoglou,I.; Panneershelvam, V.; Lanctot, M. et al. Mastering the game of Go with deep neural networks and tree search. Nature Vol. 529, No. 7587, 484-489, 2016.
Crossref Google Scholar
[70]
Nvidia. Interactive reconstruction of Monte Carlo imagesequences using a recurrent denoising autoencoder. 2020. Available at https://research.nvidia.com/publication/interactive-reconstruction-monte-carlo-image-sequences-using-recurrent-denoising.
Computational Visual Media
Volume 7 Issue 2,
June 2021
Pages 169-185
DOI: 10.1007/s41095-021-0209-9
Cite this article:
Huo Y, Yoon S-e. A survey on deep learning-based Monte Carlo denoising. Computational Visual Media, 2021, 7(2): 169-185. https://doi.org/10.1007/s41095-021-0209-9

1312

Views

76

Downloads

36

Crossref

37

Web of Science

44

Scopus

5

CSCD

Altmetrics
Received: 25 December 2020
Accepted: 23 January 2021
Published: 29 March 2021
© The Author(s) 2021

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www. editorialmanager.com/cvmj.
Return