AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Computational Visual Media Article
PDF (4.2 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Research Article | Open Access

Deep residual learning for denoising Monte Carlo renderings

Artixels, Hong Kong S.A.R., China.
Department of Computer Science and Engineering, the Chinese University of Hong Kong, Hong Kong S.A.R., China.
Show Author Information

Abstract

Learning-based techniques have recently been shown to be effective for denoising Monte Carlo rendering methods. However, there remains a quality gap to state-of-the-art handcrafted denoisers. In this paper, we propose a deep residual learning based method that outperforms both state-of-the-art handcrafted denoisers and learning-based denoisers. Unlike the indirect nature of existing learning-based methods (which e.g., estimate the parameters and kernel weights of an explicit feature based filter), we directly map the noisy input pixels to the smoothed output. Using this direct mapping formulation, we demonstrate that even a simple-and-standard ResNet and three common auxiliary features (depth, normal, and albedo) are sufficient to achieve high-quality denoising. This minimal requirement on auxiliary data simplifies both training and integration of our method into most production rendering pipelines. We have evaluated our method on unseen images created by a different renderer. Consistently superior quality denoising is obtained in all cases.

Keywords

deep learningdenoisingMonte Carlo renderingdeep residual learningfilter-free denoising

Electronic Supplementary Material

Download File(s)
41095_2019_142_MOESM1_ESM.pdf (39.6 MB)

References

[1]
W. Jakob,; S. Marschner, Manifold exploration: A Markov Chain Monte Carlo technique for rendering scenes with difficult specular transport. ACM Transac-tions on Graphics Vol. 31, No. 4 Article No. 58, 2012.
Crossref Google Scholar
[2]
T. Hachisuka,; J. Pantaleoni,; H. W. Jensen, A path space extension for robust light transport simulation. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 191, 2012.
Crossref Google Scholar
[3]
I. Georgiev,; J. Křivánek,; T. Davidovič,; P. Slusallek, Light transport simulation with vertex connection and merging. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 192, 2012.
Crossref Google Scholar
[4]
B. Moon,; N. Carr,; S. E. Yoon, Adaptive rendering based on weighted local regression. ACM Transactions on Graphics Vol. 33, No. 5, Article No. 170, 2014.
Crossref Google Scholar
[5]
B. Bitterli,; F. Rousselle,; B. Moon,; J. A. Iglesias-Guitián,; D. Adler,; K. Mitchell,; W. Jarosz,; J. NováK, Nonlinearly weighted first-order regression for denoising Monte Carlo renderings. Computer Graphics Forum Vol. 35, No. 4, 107-117, 2016.
Crossref Google Scholar
[6]
F. Rousselle,; M. Manzi,; M. Zwicker, Robust denoising using feature and color information. Computer Graphics Forum Vol. 32, No. 7, 121-130, 2013.
Crossref Google Scholar
[7]
S. Bako,; T. Vogels,; B. McWilliams,; M. Meyer,; J. NováK,; A. Harvill,; P. Sen,; T. DeRose,; F. Rousselle, Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Transactions on Graphics Vol. 36, No. 4, 1-14, 2017.
Crossref Google Scholar
[8]
N. K. Kalantari,; S. Bako,; P. Sen, A machine learning approach for filtering Monte Carlo noise. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 122, 2015.
Crossref Google Scholar
[9]
K. M. He,; X. Y. Zhang,; S. Q. Ren,; J. Sun, Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770-778, 2016.
Crossref
[10]
K. M. He,; X. Y. Zhang,; S. Q. Ren,; J. Sun, Identity mappings in deep residual networks. In: Computer Vision - ECCV 2016. Lecture Notes in Computer Science, Vol 9908. B. Leibe,; J. Matas,; N. Sebe,; M. Welling, Eds. Springer Cham, 630-645, 2016.
[11]
B. Bitterli, Rendering resources 2016. Available at https://benedikt-bitterli.me/resources/.
[12]
S. Ioffe,; C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
[13]
M. Zwicker,; W. Jarosz,; J. Lehtinen,; B. Moon,; R. Ramamoorthi,; F. Rousselle,; P. Sen,; C. Soler,; S.-E. Yoon, Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. Computer Graphics Forum Vol. 34, No. 2, 667-681, 2015.
Google Scholar
[14]
I. Goodfellow,; Y. Bengio,; A. Courville,; Y. Bengio, Deep Learning, Vol. 1. MIT Press, 2016.
[15]
Y. LeCun,; Y. Bengio,; G. Hinton, Deep learning. Nature Vol. 521, No. 7553, 436-444, 2015.
Google Scholar
[16]
J. Schmidhuber, Deep learning in neural networks: An overview. Neural Networks Vol. 61, 85-117, 2015.
Google Scholar
[17]
M. D. McCool, Anisotropic diffusion for Monte Carlo noise reduction. ACM Transactions on Graphics Vol. 18, No. 2, 171-194, 1999.
Google Scholar
[18]
P. Perona,; J. Malik, Scale-space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 12, No. 7, 629-639, 1990.
Google Scholar
[19]
H. Dammertz,; D. Sewtz,; J. Hanika,; H. P. A. Lensch, Edge-avoiding À-Trous wavelet transform for fast global illumination filtering. In: Proceedings of the Conference on High Performance Graphics, 67-75, 2010.
[20]
R. Fattal, Edge-avoiding wavelets and their applications. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 22, 2009.
Google Scholar
[21]
P. Sen,; S. Darabi, On filtering the noise from the random parameters in Monte Carlo rendering. ACM Transactions on Graphics Vol. 31, No. 3, Article No. 18, 2012.
Google Scholar
[22]
V. Aurich,; J. Weule, Non-linear Gaussian filters performing edge preserving diffusion. In: Mustererkennung 1995. Informatik aktuell. G. Sagerer,; S. Posch,; F. Kummert, Eds Springer Berlin Heidelberg, 538-545 1995.
[23]
C. Tomasi,; R. Manduchi, Bilateral filtering for gray and color images. In: Proceedings of the IEEE International Conference on Computer Vision, 839-846, 1998.
[24]
E. Eisemann,; F. Durand, Flash photography enhancement via intrinsic relighting. ACM Transactions on Graphics Vol. 23, No. 3, 673-678, 2004.
Google Scholar
[25]
T. M. Li,; Y. T. Wu,; Y. Y. Chuang, SURE-based optimization for adaptive sampling and reconstruction. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 194, 2012.
Google Scholar
[26]
C. M. Stein, Estimation of the mean of a multivariate normal distribution. The Annals of Statistics Vol. 9, No. 6, 1135-1151, 1981.
Google Scholar
[27]
D. Van de Ville,; M. Kocher, SURE-based non-local means. IEEE Signal Processing Letters Vol. 16, No. 11, 973-976, 2009.
Google Scholar
[28]
A. Buades,; B. Coll,; J. M. Morel, Nonlocal image and movie denoising. International Journal of Computer Vision Vol. 76, No. 2, 123-139, 2008.
Google Scholar
[29]
B. Moon,; J. A. Iglesias-Guitian,; S. E. Yoon,; K. Mitchell, Adaptive rendering with linear predictions. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 121, 2015.
Google Scholar
[30]
A. Krizhevsky,; I. Sutskever,; G. E. Hinton, ImageNet classification with deep convolutional neural networks. In: Proceedings of the Advances in Neural Information Processing Systems 25, 1097-1105, 2012.
[31]
C. Szegedy,; W. Liu,; Y. Q. Jia,; P. Sermanet,; S. Reed,; D. Anguelov,; D. Erhan,; V. Vanhoucke,; A. Rabinovich, Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1-9, 2015.
[32]
K. M. He,; X. Y. Zhang,; S. Q. Ren,; J. Sun, Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In: Proceedings of the IEEE International Conference on Computer Vision, 1026-1034, 2015.
[33]
J. Xie,; L. Xu,; E. Chen, Image denoising and inpainting with deep neural networks. In: Proceedings of the Advances in Neural Information Processing Systems 25, 341-349, 2012.
[34]
S. Iizuka,; E. Simo-Serra,; H. Ishikawa, Globally and locally consistent image completion. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 107, 2017.
Google Scholar
[35]
L. Xu,; J. S. Ren,; C. Liu,; J. Jia, Deep convolutional neural network for image deconvolution. In: Proceedings of the Advances in Neural Information Processing Systems 27, 1790-1798, 2014.
[36]
W. Z. Shi,; J. Caballero,; F. Huszar,; J. Totz,; A. P. Aitken,; R. Bishop,; D. Rueckert,; Z. Wang, Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1874-1883, 2016.
[37]
C. Ledig,; L. Theis,; F. Huszar,; J. Caballero,; A. Cunningham,; A. Acosta,; A. Aitken,; A. Tejani,; J. Totz,; Z. Wang,; W. Shi, Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 105-114, 2017.
[38]
K. Simonyan,; A. Zisserman, Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[39]
X. Glorot,; Y. Bengio, Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, 249-256, 2010.
[40]
S. Nah,; T. H. Kim,; K. M. Lee, Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 257-265, 2017.
[41]
R. F. Xu,; S. N. Pattanaik, A novel Monte Carlo noise reduction operator. IEEE Computer Graphics and Applications Vol. 25, No. 2, 31-35, 2005.
Google Scholar
[42]
P. Shirley,; A. L. Timo,; J. Cohen,; E. Enderton,; S. Laine,; D. Luebke,; M. McGuire, A local image reconstruction algorithm for stochastic rendering. In: Proceedings of the Symposium on Interactive 3D Graphics and Games, 9-14, 2011.
[43]
P. Bauszat,; M. Eisemann,; M. Magnor, Guided image filtering for interactive high-quality global illumination. Computer Graphics Forum Vol. 30, No. 4, 1361-1368, 2011.
Google Scholar
[44]
F. Rousselle,; C. Knaus,; M. Zwicker, Adaptive sampling and reconstruction using greedy error minimization. ACM Transactions on Graphics Vol. 30, No. 6, Article No. 159, 2011.
Google Scholar
[45]
B. Lim,; S. Son,; H. Kim,; S. Nah,; K. M. Lee, Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 1132-1140, 2017.
[46]
X. Mao,; C. Shen,; Y.-B. Yang, Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Proceedings of the Advances in Neural Information Processing Systems 29, 2802-2810, 2016.
[47]
M. D. Zeiler,; R. Fergus, Visualizing and understanding convolutional networks. In: Computer Vision - ECCV 2014. Lecture Notes in Computer Science, Vol. 8689. D. Fleet,; T. Pajdla,; B. Schiele,; T. Tuytelaars, Eds. Springer Cham, 818-833, 2014.
[48]
D. Yu,; A. Eversole,; M. L. Seltzer,; K. Yao,; Z. Huang,; B. Guenter,; O. Kuchaiev,; Y. Zhang,; F. Seide,; H. Wang, et al. An introduction to computational networks and the computational network toolkit. Microsoft Technical Report MSR-TR-2014-112. 2014.
[49]
CNTK. Microsoft cognitive toolkit. 2018.
[50]
L. Gritz, Openimageio software. 2008.
[51]
S. Van der Walt,; S. C. Colbert,; G. Varoquaux, The NumPy array: A structure for efficient numerical computation. Computing in Science & Engineering Vol. 13, No. 2, 22-30, 2011.
Google Scholar
[52]
D. P. Kingma,; J. Ba, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[53]
H. Zhao,; O. Gallo,; I. Frosio,; J. Kautz, Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging Vol. 3, No. 1, 47-57, 2017.
Google Scholar
[54]
C. R. A. Chaitanya,; A. S. Kaplanyan,; C. Schied,; M. Salvi,; A. Lefohn,; D. Nowrouzezahrai,; T. Aila, Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 98, 2017.
Google Scholar
[55]
J. Johnson,; A. Alahi,; F. F. Li, Perceptual losses for real-time style transfer and super-resolution. In: Computer Vision - ECCV 2016. Lecture Notes in Computer Science, Vol. 9906. B. Leibe,; J. Matas,; N. Sebe,; M. Welling, Eds. Springer Cham, 694-711, 2016.
[56]
R. K. Srivastava,; K. Greff,; J. Schmidhuber Highway networks. arXiv preprint arXiv:1505.00387, 2015.
[57]
G. Huang,; Y. Sun,; Z. Liu,; D. Sedra,; K. Q. Weinberger, Deep networks with stochastic depth. In: Computer Vision - ECCV 2016. Lecture Notes in Computer Science, Vol 9908. B. Leibe,; J. Matas,; N. Sebe,; M. Welling, Eds. Springer Cham, 646-661, 2016.
[58]
S. Zagoruyko,; N. Komodakis, Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.
[59]
Z. Wang,; A. C. Bovik,; H. R. Sheikh,; E. P. Simoncelli, Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing Vol. 13, No. 4, 600-612, 2004.
Google Scholar
[60]
W. L. Briggs,; S. F. McCormick, A Multigrid Tutorial, Vol. 72. Siam, 2000.
[61]
H. Zimmer,; F. Rousselle,; W. Jakob,; O. Wang,; D. Adler,; W. Jarosz,; O. Sorkine-Hornung,; A. Sorkine-Hornung, Path-space motion estimation and decomposition for robust animation filtering. Computer Graphics Forum Vol. 34, No. 4, 131-142, 2015.
Google Scholar
Computational Visual Media
Volume 5 Issue 3,
September 2019
Pages 239-255
DOI: 10.1007/s41095-019-0142-3
Cite this article:
Wong K-M, Wong T-T. Deep residual learning for denoising Monte Carlo renderings. Computational Visual Media, 2019, 5(3): 239-255. https://doi.org/10.1007/s41095-019-0142-3

684

Views

21

Downloads

15

Crossref

N/A

Web of Science

18

Scopus

3

CSCD

Altmetrics
Revised: 12 March 2019
Accepted: 14 April 2019
Published: 09 May 2019
© The author(s) 2019

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from thecopyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
Return