AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Computational Visual Media Article
PDF (2.6 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Research Article | Open Access

NPRportrait 1.0: A three-level benchmark for non-photorealistic rendering of portraits

School of Computer Science and Informatics, Cardiff University, Cardiff, UK
School of Computer Science, Carleton University, Ottawa, Canada
Department of Computer Science and Technology, Tsinghua University, Beijing, China
Reichman University (the Interdisciplinary Center), Herzliya, Israel
Department of Computer Science and Engineering, Pohang University of Science and Technology, Pohang, Republic of Korea
Lambda Labs, Inc., San Francisco, USA
Hasso Plattner Institute, University of Potsdam, Potsdam, Germany
Multimedia Processing Laboratory, Samsung Advanced Institute of Technology, Suwon, Republic of Korea
Adobe Systems, Inc., San Jose, USA
Show Author Information

Graphical Abstract

Abstract

Recently, there has been an upsurge of activity in image-based non-photorealistic rendering (NPR), and in particular portrait image stylisation, due to the advent of neural style transfer (NST). However, the state of performance evaluation in this field is poor, especially compared to the norms in the computer vision and machine learning communities. Unfortunately, thetask of evaluating image stylisation is thus far not well defined, since it involves subjective, perceptual, and aesthetic aspects. To make progress towards a solution, this paper proposes a new structured, three-level, benchmark dataset for the evaluation of stylised portrait images. Rigorous criteria were used for its construction, and its consistency was validated by user studies. Moreover, a new methodology has been developed for evaluating portrait stylisation algorithms, which makes use of the different benchmark levels as well as annotations provided by user studies regarding the characteristics of the faces. We perform evaluation for a wide variety of image stylisation methods (both portrait-specific and general purpose, and also both traditional NPR approaches and NST) using the new benchmark dataset.

Keywords

non-photorealistic rendering (NPR)imagestylizationstyle transferportraitevaluationbenchmark

Electronic Supplementary Material

Download File(s)
101320TP-2022-3-445_ESM.pdf (15.8 MB)

References

[1]
Kyprianidis, J. E.; Collomosse, J.; Wang, T. H.; Isenberg, T. State of the “art”: A taxonomy of artistic stylization techniques for images and video. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 5, 866-885, 2013.
Crossref Google Scholar
[2]
Rosin, P.; Collomosse, J. Image and Video-Based Artistic Stylisation. London: Springer London, 2013.
Crossref
[3]
Gatys, L. A.; Ecker, A. S.; Bethge, M. Image styletransfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2414-2423, 2016.
Crossref
[4]
Jing, Y. C.; Yang, Y. Z.; Feng, Z. L.; Ye, J. W.; Yu, Y. Z.; Song, M. L. Neural style transfer: A review. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 11, 3365-3385, 2020.
Crossref Google Scholar
[5]
Semmo, A.; Isenberg, T.; Döllner, J. Neural style transfer: A paradigm shift for image-based artistic rendering? In: Proceedings of the Symposium on Non-Photorealistic Animation and Rendering, Article No. 5, 2017.
Crossref
[6]
Gooch, A. A.; Long, J.; Ji, L.; Estey, A.; Gooch, B. S. Viewing progress in non-photorealistic rendering through Heinlein’s lens. In: Proceedings of the 8thInternational Symposium on Non-Photorealistic Animation and Rendering, 165-171, 2010.
Crossref
[7]
Hall, P.; Lehmann, A.-S. Don’t measure—Appreciate! NPR seen through the prism of art history. In: Image and Video-Based Artistic Stylisation. Computational Imaging and Vision, Vol. 42. Rosin, P.; Collomosse, J. Eds. Springe London, 333-351, 2013.
Crossref
[8]
Mould, D.; Rosin, P. L. Developing and applying a benchmark for evaluating image stylization. Computers & Graphics Vol. 67, 58-76, 2017.
Crossref Google Scholar
[9]
Rosin, P. L.; Mould, D.; Berger, I.; Collomosse, J.; Lai, Y.; Li, C.; Li, H.; Shamir, A.; Wand, M.; Wang, T.; et al. Benchmarking non-photorealistic rendering of portraits. In: Proceedings of the Symposium on Non-Photorealistic Animation and Rendering, Article No. 11, 2017.
Crossref
[10]
Fisher, R. B. CVonline. Available at http://homepages.inf.ed.ac.uk/rbf/CVonline.
[11]
Kumar, M. P. P.; Poornima, B.; Nagendraswamy, H. S.; Manjunath, C. A comprehensive survey on non-photorealistic rendering and benchmark developments for image abstraction and stylization. Iran Journal of Computer Science Vol. 2, No. 3, 131-165, 2019.
Crossref Google Scholar
[12]
Buolamwini, J.; Gebru, T. Gender shades: Intersectional accuracy disparities in commercial gender classification. In: Proceedings of the Conference on Fairness, Accountability and Transparency, 77-91, 2018.
[13]
Azami, R.; Mould, D. Detail and color enhancement in photo stylization. In: Proceedings of the Symposium on Computational Aesthetics, Article No. 5, 2017.
Crossref
[14]
Du, L. How much deep learning does neural style transfer really need? An ablation study. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 3139-3148, 2020.
Crossref
[15]
Rosin, P. L.; Lai, Y. K. Watercolour rendering of portraits. In: Image and Video Technology. Lecture Notes in Computer Science, Vol. 10799. Satoh, S. Ed. Springer Cham, 268-282, 2018.
Crossref
[16]
Wu, T.; Chen, X.; Lu, L. Q. Field coupling-based image filter for sand painting stylization. Mathematical Problems in Engineering Vol. 2018, 3670498, 2018.
Crossref Google Scholar
[17]
Low, P. E.; Wong, L. K.; See, J.; Ng, R. Pic2PolyArt: Transforming a photograph into polygon-based geometric art. Signal Processing: Image Communication Vol. 91, 116090, 2021.
Crossref Google Scholar
[18]
Meier, P.; Lohweg, V. Content representation for neural style transfer algorithms based on structural similarity. In: Proceedings of the Computational Intelligence Workshop, 2019.
[19]
Shen, Q.; Zou, L.; Wang, F. J.; Huang, Z. J. A scale-adaptive color preservation neural style transfer method. In: Proceedings of the 5th International Conference on Mathematics and Artificial Intelligence, 5-9, 2020.
Crossref
[20]
Klingbeil, M.; Pasewaldt, S.; Semmo, A.; Döllner, J. Challenges in user experience design of image filtering apps. In: Proceedings of the SIGGRAPH Asia Mobile Graphics & Interactive Applications, Article No. 22, 2017.
Crossref
[21]
Trapp, M.; Pasewaldt, S.; Dürschmid, T.; Semmo, A.; Döllner, J. Teaching image-processing programming for mobile devices: A software development perspective. In: Proceedings of the Annual European Association for Computer Graphics Conference: Education Papers, 17-24, 2018.
[22]
Wang, Z.; Bovik, A. C.; Sheikh, H. R.; Simoncelli, E. P. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing Vol. 13, No. 4, 600-612, 2004.
Crossref Google Scholar
[23]
Zhang, R.; Isola, P.; Efros, A. A.; Shechtman, E.; Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 586-595, 2018.
Crossref
[24]
Zamir, S. W.; Vazquez-Corral, J.; Bertalmío, M. Vision models for wide color gamut imaging in cinema. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 43, No. 5, 1777-1790, 2019.
Crossref Google Scholar
[25]
Kettunen, M.; Härkönen, E.; Lehtinen, J. E-LPIPS: Robust perceptual image similarity via randomtransformation ensembles. arXiv preprint arXiv:1906.03973, 2019.
Google Scholar
[26]
Moorthy, A. K.; Bovik, A. C. Blind image quality assessment: From natural scene statistics to perceptual quality. IEEE Transactions on Image Processing Vol. 20, No. 12, 3350-3364, 2011.
Crossref Google Scholar
[27]
Mittal, A.; Moorthy, A. K.; Bovik, A. C. No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing Vol. 21, No. 12, 4695-4708, 2012.
Crossref Google Scholar
[28]
Zhang, L.; Zhang, L.; Bovik, A. C. A feature-enriched completely blind image quality evaluator. IEEE Transactions on Image Processing Vol. 24, No. 8, 2579-2591, 2015.
Crossref Google Scholar
[29]
Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; Hochreiter, S. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, 6629-6640, 2017.
[30]
Bińkowski, M.; Sutherland, D. J.; Arbel, M.; Gretton, A. Demystifying MMD GANs. In: Proceedings of the 6th International Conference on Learning Representations, 2018.
[31]
Isenberg, T. Evaluating and validating non-photorealistic and illustrative rendering. In: Image and Video-Based Artistic Stylisation. Computational Imaging and Vision, Vol. 42. Rosin, P.; Collomosse, J. Eds. Springer London, 311-331, 2013.
Crossref
[32]
Hertzmann, A. Non-Photorealistic Rendering and the science of art. In: Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering, 147-157, 2010.
Crossref
[33]
Mould, D. Authorial subjective evaluation of non-photorealistic images. In: Proceedings of the Workshop on Non-Photorealistic Animation and Rendering, 49-56, 2014.
Crossref
[34]
Li, Y. Z.; Kobatake, H. Extraction of facial sketch images and expression transformation based on FACS. In: Proceedings of the International Conference on Image Processing, 520-523, 1995.
[35]
Yaniv, J.; Newman, Y.; Shamir, A. The face of art: Landmark detection and geometric style in portraits. ACM Transactions on Graphics Vol. 38, No. 4, Article No. 60, 2019.
Crossref Google Scholar
[36]
Zhao, M.; Zhu, S.-C. Artistic rendering of portraits. In: Image and Video-Based Artistic Stylisation. Computational Imaging and Vision, Vol. 42. Rosin, P.; Collomosse, J. Eds. Springer London, 237-253, 2013.
Crossref
[37]
Li, C.; Wand, M. Combining Markov random fields and convolutional neural networks for image synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2479-2486, 2016.
Crossref
[38]
Berger, I.; Shamir, A.; Mahler, M.; Carter, E.; Hodgins, J. Style and abstraction in portrait sketching. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 55, 2013.
Crossref Google Scholar
[39]
Yi, R.; Liu, Y. J.; Lai, Y. K.; Rosin, P. L. APDrawingGAN: Generating artistic portrait drawings from face photos with hierarchical GANs. In: Pro-ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10735-10744, 2019.
Crossref
[40]
Rosin, P. L.; Lai, Y.-K. Non-photorealistic rendering of portraits. In: Proceedings of the workshop on Computational Aesthetics, 159-170, 2015.
[41]
Winnemöller, H.; Kyprianidis, J. E.; Olsen, S. C. XDoG: An eXtended difference-of-Gaussians compendium including advanced image stylization. Computers & Graphics Vol. 36, No. 6, 740-753, 2012.
Crossref Google Scholar
[42]
Rosin, P. L.; Lai, Y. K. Image-based portrait engraving. arXiv preprint arXiv:2008.05336, 2020.
Google Scholar
[43]
Son, M.; Lee, Y. J.; Kang, H.; Lee, S. Structure grid for directional stippling. Graphical Models Vol. 73, No. 3, 74-87, 2011.
Crossref Google Scholar
[44]
Semmo, A.; Limberger, D.; Kyprianidis, J. E.; Döllner, J. Image stylization by interactive oil paint filtering. Computers & Graphics Vol. 55, 157-171, 2016.
Crossref Google Scholar
[45]
Doyle, L.; Anderson, F.; Choy, E.; Mould, D. Automated pebble mosaic stylization of images. Computational Visual Media Vol. 5, No. 1, 33-44, 2019.
Crossref Google Scholar
[46]
Bruce, V.; Young, A. Face Perception. Psychology Press, 2013.
Crossref
[47]
Van Koppen, P. J.; Lochun, S. K. Portraying perpetrators: The validity of offender descriptions by witnesses. Law and Human Behavior Vol. 21, No. 6, 661-685, 1997.
Crossref Google Scholar
[48]
Fahsing, I. A.; Ask, K.; Granhag, P. A. The man behind the mask: Accuracy and predictors of eyewitness offender descriptions. Journal of Applied Psychology Vol. 89, No. 4, 722-729, 2004.
Crossref Google Scholar
[49]
Dobs, K.; Isik, L.; Pantazis, D.; Kanwisher, N. How face perception unfolds over time. Nature Communications Vol. 10, No. 1, 1258, 2019.
Crossref Google Scholar
[50]
Wheeler, B. AlgDesign: Algorithmic experimental design. R package version 1.1-7. 2014. Available at https://cran.rproject.org/web/packages/AlgDesign/.
[51]
Atkinson, A.; Donev, A.; Tobias, R. Optimum Experimental Designs, with SAS, Volume 34. Oxford University Press, 2007.
[52]
Fedorov, V. Theory of Optimal Experiments. Academic Press, 1972.
[53]
Doyle, R. Ethnic groups in the world. Scientific American Vol. 279, No. 3, 30, 1998.
Crossref Google Scholar
[54]
McLellan, B.; McKelvie, S. J. Effects of age and gender on perceived facial attractiveness. Canadian Journal of Behavioural Science/Revue Canadienne des Sciences du Comportement Vol. 25, No. 1, 135-142, 1993.
Crossref Google Scholar
[55]
Batres, C.; Kannan, M.; Perrett, D. I. Familiarity with own population’s appearance influences facial preferences. Human Nature Vol. 28, No. 3, 344-354, 2017.
Crossref Google Scholar
[56]
Cooper, P. A.; Maurer, D. The influence of recent experience on perceptions of attractiveness. Perception Vol. 37, No. 8, 1216-1226, 2008.
Crossref Google Scholar
[57]
Sanakoyeu, A.; Kotovenko, D.; Lang, S.; Ommer, B. A style-aware content loss for real-time HD style transfer. In: Computer Vision - ECCV 2018. Lecture Notes in Computer Science, Vol. 11212. Ferrari, V.; Hebert M.; Sminchisescu C.; Weiss, Y. Eds. Springer Cham, 698-714, 2018.
[58]
Cha, S. H.; Srihari, S. N. On measuring the distance between histograms. Pattern Recognition Vol. 35, No. 6, 1355-1370, 2002.
Crossref Google Scholar
[59]
Wauthier, F. L.; Jordan, M. I.; Jojic, N. Efficient ranking from pairwise comparisons. In: Proceedings of the 30th International Conference on Machine Learning, Vol. 28, III-109-III-117, 2013.
Computational Visual Media
Volume 8 Issue 3,
September 2022
Pages 445-465
DOI: 10.1007/s41095-021-0255-3
Cite this article:
Rosin PL, Lai Y-K, Mould D, et al. NPRportrait 1.0: A three-level benchmark for non-photorealistic rendering of portraits. Computational Visual Media, 2022, 8(3): 445-465. https://doi.org/10.1007/s41095-021-0255-3

946

Views

31

Downloads

7

Crossref

4

Web of Science

24

Scopus

0

CSCD

Altmetrics
Received: 17 July 2021
Accepted: 16 September 2021
Published: 06 April 2022
© The Author(s) 2021.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www. editorialmanager.com/cvmj.
Return