PDF (7.7 MB)
Collect
Submit Manuscript
Research Article | Open Access

DualSmoke: Sketch-based smoke illustration design with two-stage generative model

Graduate School of Advanced Science and Technology, Japan Advanced Institute of Science and Technology, 1-1 Nomi, Ishikawa 9231292, Japan
Graduate School of Computer and Information Sciences, Hosei University, 3-7-2 Kajino-cho, Koganei-shi, Tokyo 1848584, Japan
Show Author Information

Graphical Abstract

View original image Download original image

Abstract

The dynamic effects of smoke are impressive in illustration design, but it is a troublesome and challenging issue for inexpert users to design smoke effects without domain knowledge of fluid simulations. In this work, we propose DualSmoke, a two-stage global-to-local generation framework for interactive smoke illustration design. In the global stage, the proposed approach utilizes fluid patterns to generate Lagrangian coherent structures from the user's hand-drawn sketches. In the local stage, detailed flow patterns are obtained from the generated coherent structure. Finally, we apply a guiding force field to the smoke simulator to produce the desired smoke illustration. To construct the training dataset, DualSmoke generates flow patterns using finite-time Lyapunov exponents of the velocity fields. The synthetic sketch data are generated from the flow patterns by skeleton extraction. Our user study verifies that the proposed design interface can provide various smoke illustration designs with good user usability. Our code is available at https://github.com/shasph/DualSmoke.

Electronic Supplementary Material

Video
cvm-10-5-965-ESM.mp4

References

[1]
Kazi, R. H.; Chevalier, F.; Grossman, T.; Zhao, S. D.; Fitzmaurice, G. Draco: Bringing life to illustrations with kinetic textures. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 351–360, 2014.
[2]
Xing, J.; Kazi, R. H.; Grossman, T.; Wei, L. Y.; Stam, J.; Fitzmaurice, G. Energy-brushes: Interactive tools for illustrating stylized elemental dynamics. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, 755–766, 2016.
[3]

Zhu, B.; Iwata, M.; Haraguchi, R.; Ashihara, T.; Umetani, N.; Igarashi, T.; Nakazawa, K. Sketch-based dynamic illustration of fluid systems. ACM Transactions on Graphics Vol. 30, No. 6, 1–8, 2011.

[4]

Hu, Z. Y.; Xie, H. R.; Fukusato, T.; Sato, T.; Igarashi, T. Sketch2VF: Sketch-based flow design with conditional generative adversarial network. Computer Animation and Virtual Worlds Vol. 30, Nos. 3–4, e1889, 2019.

[5]
Rasmussen, N.; Enright, D.; Nguyen, D.; Marino, S.; Sumner, N.; Geiger, W.; Hoon, S.; Fedkiw, R. Directable photorealistic liquids. In: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 193–202, 2004.
[6]

Treuille, A.; McNamara, A.; Popović, Z.; Stam, J. Keyframe control of smoke simulations. ACM Transactions on Graphics Vol. 22, No. 3, 716–723, 2003.

[7]

Nielsen, M. B.; Christensen, B. B. Improved variational guiding of smoke animations. Computer Graphics Forum Vol. 29, No. 2, 705–712, 2010.

[8]

Sato, S.; Dobashi, Y.; Kim, T. Stream-guided smoke simulations. ACM Transactions on Graphics Vol. 40, No. 4, Article No. 161, 2021.

[9]
Kim, Y.; Machiraju, R.; Thompson, D. Path-based control of smoke simulations. In: Proceedings of the ACM SIGGRAPH/Eurographics symposium on Computer animation, 33–42, 2006.
[10]

Kim, B.; Huang, X. C.; Wuelfroth, L.; Tang, J. W.; Cordonnier, G.; Gross, M.; Solenthaler, B. Deep reconstruction of 3D smoke densities from artist sketches. Computer Graphics Forum Vol. 41, No. 2, 97–110, 2022.

[11]

Bhattacharjee, S.; Chaudhuri, P. A survey on sketch based content creation: From the desktop to virtual and augmented reality. Computer Graphics Forum Vol. 39, No. 2, 757–780, 2020.

[12]
Igarashi, T.; Matsuoka, S.; Tanaka, H. Teddy: A sketching interface for 3D freeform design. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, 409–416, 1999.
[13]

Guérin, É.; Digne, J.; Galin, É.; Peytavie, A.; Wolf, C.; Benes, B.; Martinez, B. Interactive example-based terrain authoring with conditional generative adversarial networks. ACM Transactions on Graphics Vol. 36, No. 6, Article No. 228, 2017.

[14]

Brodt, K.; Bessmeltsev, M. Sketch2Pose: Estimating a 3D character pose from a bitmap sketch. ACM Transactions on Graphics Vol. 41, No. 4, Article No. 85, 2022.

[15]

Ren, B.; Huang, J. H.; Lin, M. C.; Hu, S. M. Controllable dendritic crystal simulation using orientation field. Computer Graphics Forum Vol. 37, No. 2, 485–495, 2018.

[16]

Guay, M.; Ronfard, R.; Gleicher, M.; Cani, M. P. Space-time sketching of character animation. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 118, 2015.

[17]
Wu, L. Q.; Liu, S. G.; Yu, Z. J.; Sun, H. Q.; Zhang, M. Dynamic fluids mixed with local-control effects. In: Proceedings of the 7th International Conference on Image and Graphics, 694–700, 2013.
[18]

Yan, G. W.; Chen, Z. L.; Yang, J. M.; Wang, H. M. Interactive liquid splash modeling by user sketches. ACM Transactions on Graphics Vol. 39, No. 6, Article No. 165, 2020.

[19]

Huang, Z. Y.; Peng, Y. C.; Hibino, T.; Zhao, C. Q.; Xie, H. R.; Fukusato, T.; Miyata, K. DualFace: Two-stage drawing guidance for freehand portrait sketching. Computational Visual Media Vol. 8, No. 1, 63–77, 2022.

[20]
Peng, Y. C.; Zhao, C. Q.; Huang, Z. Y.; Fukusato, T.; Xie, H. R.; Miyata, K. Two-stage motion editing interface for character animation. In: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 1–2, 2021.
[21]
Stam, J. Stable fluids. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, 121–128, 1999.
[22]
Fedkiw, R.; Stam, J.; Jensen, H. W. Visual simulation of smoke. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, 15–22, 2001.
[23]

Selle, A.; Rasmussen, N.; Fedkiw, R. A vortex particle method for smoke, water and explosions. ACM Transactions on Graphics Vol. 24, No. 3, 910–914, 2005.

[24]

Dobashi, Y.; Kusumoto, K.; Nishita, T.; Yamamoto, T. Feedback control of cumuliform cloud formation based on computational fluid dynamics. ACM Transactions on Graphics Vol. 27, No. 3, 1–8, 2008.

[25]

Tang, J. W.; Azevedo, V. C.; Cordonnier, G.; Solenthaler, B. Honey, I shrunk the domain: Frequency-aware force field reduction for efficient fluids optimization. Computer Graphics Forum Vol. 40, No. 2, 339–353, 2021.

[26]

Yuan, Z.; Chen, F.; Zhao, Y. Pattern-guided smoke animation with Lagrangian coherent structure. ACM Transactions on Graphics Vol. 30, No. 6, 1–8, 2011.

[27]
Huang, R. G.; Melek, Z.; Keyser, J. Preview-based sampling for controlling gaseous simulations. In: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 177–186, 2011.
[28]

Ladický, L.; Jeong, S.; Solenthaler, B.; Pollefeys, M.; Gross, M. Data-driven fluid simulations using regression forests. ACM Transactions on Graphics Vol. 34, No. 6, Article No. 199, 2015.

[29]

Chu, M. Y.; Thuerey, N. Data-driven synthesis of smoke flows with CNN-based feature descriptors. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 69, 2017.

[30]

Xie, Y.; Franz, E.; Chu, M. Y.; Thuerey, N. tempoGAN: A temporally coherent, volumetric GAN for super-resolution fluid flow. ACM Transactions on Graphics Vol. 37, No. 4, Article No. 95, 2018.

[31]

Chu, M. Y.; Thuerey, N.; Seidel, H. P.; Theobalt, C.; Zayer, R. Learning meaningful controls for fluids. ACM Transactions on Graphics Vol. 40, No. 4, Article No. 100, 2021.

[32]

Selle, A.; Fedkiw, R.; Kim, B.; Liu, Y. J.; Rossignac, J. An unconditionally stable MacCormack method. Journal of Scientific Computing Vol. 35, Nos. 2–3, 350–371, 2008.

[33]

Shadden, S. C.; Lekien, F.; Marsden, J. E. Definition and properties of Lagrangian coherent structures from finite-time Lyapunov exponents in two-dimensional aperiodic flows. Physica D: Nonlinear Phenomena Vol. 212, Nos. 3–4, 271–304, 2005.

[34]

Ferstl, F.; Burger, K.; Theisel, H.; Westermann, R. Interactive separating streak surfaces. IEEE Transactions on Visualization and Computer Graphics Vol. 16, No. 6, 1569–1577, 2010.

[35]

Tagliasacchi, A.; Alhashim, I.; Olson, M.; Zhang, H. Mean curvature skeletons. Computer Graphics Forum Vol. 31, No. 5, 1735–1744, 2012.

[36]
Au, O. K. C.; Tai, C. L.; Chu, H. K.; Cohen-Or, D.; Lee, T. Y. Skeleton extraction by mesh contraction. In: Proceedings of the ACM SIGGRAPH Papers, 1–10, 2008.
[37]

Gao, F. Y.; Wei, G. S.; Xin, S. Q.; Gao, S. S.; Zhou, Y. F. 2D skeleton extraction based on heat equation. Computers & Graphics Vol. 74, 99–108, 2018.

[38]
Isola, P.; Zhu, J. Y.; Zhou, T. H.; Efros, A. A. Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5967–5976, 2017.
[39]
Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Lecture Notes in Computer Science, Vol. 9351. Navab, N.; Hornegger, J.; Wells, W.; Frangi, A. Eds. Springer Cham, 234–241, 2015.
[40]
Pfaff, T.; Thuerey, N. mantaflow. Available at http://mantaflow.com/
[41]
Brooke, J. SUS: A 'quick and dirty' usability scale. In: Usability Evaluation in Industry. CRC Press, 207–212, 1996.
[42]

Hart, S. G.; Staveland, L. E. Development of NASA-TLX (task load index): Results of empirical and theoretical research. Advances in Psychology Vol. 52, 139–183, 1988.

[43]

Bangor, A.; Kortum, P.; Miller, J. T. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies Vol. 4, No. 3, 114–123, 2009.

[44]

Su, W. C.; Du, D.; Yang, X.; Zhou, S. Z.; Fu, H. B. Interactive sketch-based normal map generation with deep neural networks. Proceedings of the ACM on Computer Graphics and Interactive Techniques Vol. 1, No. 1, Article No. 22, 2018.

[45]
He, Y.; Xie, H. R.; Zhang, C.; Yang, X.; Miyata, K. Sketch-based normal map generation with geometric sampling. In: Proceedings of the SPIE 11766, International Workshop on Advanced Imaging Technology, 261–266, 2021.
[46]

Garth, C.; Gerhardt, F.; Tricoche, X.; Hans, H. Efficient computation and visualization of coherent structures in fluid flow applications. IEEE Transactions on Visualization and Computer Graphics Vol. 13, No. 6, 1464–1471, 2007.

Computational Visual Media
Pages 965-979
Cite this article:
Xie H, Arihara K, Sato S, et al. DualSmoke: Sketch-based smoke illustration design with two-stage generative model. Computational Visual Media, 2024, 10(5): 965-979. https://doi.org/10.1007/s41095-022-0318-0
Metrics & Citations  
Article History
Copyright
Rights and Permissions
Return