Home Computational Visual Media Article
PDF (17 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript
Research Article | Open Access

Co-occurrence based texture synthesis

Tel Aviv University, Tel Aviv 6997801, Israel
Cornell-Tech, Cornell University, NYC, NY, 10044, USA
Show Author Information

Graphical Abstract

View original image Download original image

Abstract

As image generation techniques mature, there is a growing interest in explainable representations that are easy to understand and intuitive to manipulate. In this work, we turn to co-occurrence statistics, which have long been used for texture analysis, to learn a controllable texture synthesis model. We propose a fully convolutional generative adversarial network, conditioned locally on co-occurrence statistics, to generate arbitrarily large images while having local, interpretable control over texture appearance. To encourage fidelity to the input condition, we introduce a novel differentiable co-occurrence loss that is integrated seamlessly into our framework in an end-to-end fashion. We demonstrate that our solution offers a stable, intuitive, and interpretable latent representation for texture synthesis, which can be used to generate smooth texture morphs between different textures. We further show an interactive texture tool that allows a user to adjust local characteristics of the synthesized texture by directly using the co-occurrence values.

Keywords

co-occurrencetexture synthesisdeep learninggenerative adversarial networks (GANs)

Electronic Supplementary Material

Download File(s)
101320TP-2022-2-289_ESM.pdf (10.9 MB)

References

[1]
Radford, A.; Metz, L.; Chintala, S. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
Google Scholar
[2]
Bau, D.; Zhu, J. Y.; Strobelt, H.; Zhou, B. L.; Tenenbaum, J. B.; Freeman, W. T.; Torralba, A. Visualizing and understanding generative adversarial networks. arXiv preprint arXiv:1901.09887, 2019.
Google Scholar
[3]
Shen, Y. J.; Gu, J. J.; Tang, X. O.; Zhou, B. L. Interpreting the latent space of GANs for semantic face editing. arXiv preprint arXiv:1907.10786, 2019.
Crossref Google Scholar
[4]
Julesz, B. Visual pattern discrimination. IEEE Transactions on Information Theory Vol. 8, No. 2,84-92, 1962.
Crossref Google Scholar
[5]
Haralick, R. M.; Shanmugam, K.; Dinstein, I. Textural features for image classification. IEEE Transactions on Systems, Man, and Cybernetics Vol. SMC-3, No. 6, 610-621, 1973.
Crossref Google Scholar
[6]
Isola, P.; Zoran, D.; Krishnan, D.; Adelson, E. H. Crisp boundary detection using pointwise mutual information. In: Computer Vision-ECCV 2014. Lecture Notes in Computer Science, Vol. 8691. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer Cham,799-814, 2014.
Crossref
[7]
Heeger, D. J.; Bergen, J. R. Pyramid-based texture analysis/synthesis. In: Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, 229-238, 1995.
Crossref
[8]
De Bonet, J. S. Multiresolution sampling procedure for analysis and synthesis of texture images. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, 361-368, 1997.
Crossref
[9]
Portilla, J.; Simoncelli, E. P. A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision Vol. 40, No. 1, 49-70, 2000.
Google Scholar
[10]
Efros, A. A.; Freeman, W. T. Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, 341-346, 2001.
Crossref
[11]
Kwatra, V.; Schödl, A.; Essa, I.; Turk, G.; Bobick, A. Graphcut textures. ACM Transactions on GraphicsVol. 22, No. 3, 277-286, 2003.
Crossref Google Scholar
[12]
Kwatra, V.; Essa, I.; Bobick, A.; Kwatra, N. Texture optimization for example-based synthesis. ACM Transactions on Graphics Vol. 24, No. 3, 795-802, 2005.
Crossref Google Scholar
[13]
Wexler, Y.; Shechtman, E.; Irani, M. Space-time completion of video. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 29, No. 3,463-476, 2007.
Crossref Google Scholar
[14]
Simakov, D.; Caspi, Y.; Shechtman, E.; Irani, M. Summarizing visual data using bidirectional similarity. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1-8, 2008.
Crossref
[15]
Barnes, C.; Zhang, F. L. A survey of the state-of-the-art in patch-based synthesis. Computational Visual Media Vol. 3, No. 1, 3-20, 2017.
Crossref Google Scholar
[16]
Matusik, W.; Zwicker, M.; Durand, F. Texture design using a simplicial complex of morphable textures. ACM Transactions on Graphics Vol. 24, No. 3, 787-794, 2005.
Crossref Google Scholar
[17]
Rabin, J.; Peyré, G.; Delon, J.; Bernot, M. Wasserstein barycenter and its application to texture mixing. In: Scale Space and Variational Methods in Computer Vision. Lecture Notes in Computer Science, Vol. 6667. Bruckstein, A. M.; ter Haar Romeny, B. M.; Bronstein, A. M.; Bronstein, M. M. Eds. Springer Berlin Heidelberg, 435-446, 2012.
Crossref
[18]
Darabi, S.; Shechtman, E.; Barnes, C.; Goldman, D. B.; Sen, P. Image melding: Combining inconsistent images using patch-based synthesis. ACM Transactions on Graphics Vol. 31, No. 4, Article No. 82, 2012.
Crossref Google Scholar
[19]
Gatys, L. A.; Ecker, A. S.; Bethge, M. Texture synthesis using convolutional neural networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, 262-270, 2015.
Crossref
[20]
Ulyanov, D.; Lebedev, V.; Vedaldi, A.; Lempitsky, V. S. Texture networks: Feed-forward synthesis of textures and stylized images. In: Proceedings of the International Conference on Machine Learning, 1349-1357, 2016.
[21]
Sendik, O.; Cohen-Or, D. Deep correlations for texture synthesis. ACM Transactions on Graphics Vol. 36,No. 5, Article No. 161, 2017.
Crossref Google Scholar
[22]
Li, C.; Wand, M. Precomputed real-time texture synthesis with Markovian generative adversarial networks. In: Computer Vision-ECCV 2016. Lecture Notes in Computer Science, Vol. 9907. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 702-716, 2016.
Crossref
[23]
Gang, L.; Gousseau, Y.; Xia, G. S. Texture synthesis through convolutional neural networks and spectrum constraints. In: Proceedings of the 23rd International Conference on Pattern Recognition, 3234-3239, 2016.
[24]
Li, Y. J.; Fang, C.; Yang, J. M.; Wang, Z. W.; Lu, X.; Yang, M. H. Diversified texture synthesiswith feed-forward networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 266-274, 2017.
[25]
Zhou, Y.; Zhu, Z.; Bai, X.; Lischinski, D.; Cohen-Or, D.; Huang, H. Non-stationary texture synthesis by adversarial expansion. ACM Transactions on Graphics Vol. 37, No. 4, Article No. 49, 2018.
Crossref Google Scholar
[26]
Frühstück, A.; Alhashim, I.; Wonka, P. TileGAN: Synthesis of large-scale non-homogeneous textures. ACM Transactions on Graphics Vol. 38, No. 4, Article No. 58, 2019.
Crossref Google Scholar
[27]
Jetchev, N.; Bergmann, U.; Vollgraf, R. Texture synthesis with spatial generative adversarial networks. In: Proceedings of the Workshop on Adversarial Training, 2016.
[28]
Bergmann, U.; Jetchev, N.; Vollgraf, R. Learning texture manifolds with the periodic spatial GAN. arXiv preprint arXiv:1705.06566, 2017.
Google Scholar
[29]
Yu, N.; Barnes, C.; Shechtman, E.; Amirghodsi, S.; Lukác, M. Texture mixer: A network for controllable synthesis and interpolation of texture. arXiv preprint arXiv:1901.03447, 2019.
Crossref Google Scholar
[30]
Yellott, J. I. Implications of triple correlation uniqueness for texture statistics and the Julesz conjecture. Journal of the Optical Society of America A Vol. 10, No. 5, 777-793, 1993.
Crossref Google Scholar
[31]
Jevnisek, R. J.; Avidan, S. Co-occurrence filter. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3184-3192, 2017.
Crossref
[32]
Kat, R.; Jevnisek, R.; Avidan, S. Matching pixels using co-occurrence statistics. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1751-1759, 2018.
Crossref
[33]
Gulrajani, I.; Ahmed, F.; Arjovsky, M.; Dumoulin, V.; Courville, A. C. Improved training of Wasserstein GANs. In: Proceedings of the Annual Conference on Neural Information Processing Systems, 2017.
[34]
Cimpoi, M.; Maji, S.; Kokkinos, I.; Mohamed, S.; Vedaldi, A. Describing textures in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3606-3613, 2014.
Crossref
[35]
Bellini, R.; Kleiman, Y.; Cohen-Or, D. Time-varying weathering in texture space. ACM Transactions on Graphics Vol. 35, No. 4, Article No. 141, 2016.
Crossref Google Scholar
[36]
Wu, F. Z.; Dong, W. M.; Kong, Y.; Mei, X.; Yan, D. M.; Zhang, X. P.; Paul, J.-C. Feature-aware natural texture synthesis. The Visual Computer Vol. 32, No. 1, 43-55, 2016.
Crossref Google Scholar
[37]
Kaspar, A.; Neubert, B.; Lischinski, D.; Pauly, M.; Kopf, J. Self tuning texture optimization. Computer Graphics Forum Vol. 34, No. 2, 349-359, 2015.
Crossref Google Scholar
Computational Visual Media
Volume 8 Issue 2,
June 2022
Pages 289-302
DOI: 10.1007/s41095-021-0243-7
Cite this article:
Darzi A, Lang I, Taklikar A, et al. Co-occurrence based texture synthesis. Computational Visual Media, 2022, 8(2): 289-302. https://doi.org/10.1007/s41095-021-0243-7
Metrics & Citations  
Article History
Copyright
Rights and Permissions
Return