AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Computational Visual Media Article
PDF (11.4 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Research Article | Open Access

Polygonal finite element-based content-aware image warping

School of Mathematical Sciences, Xiamen University, Xiamen361005, China
Fujian Provincial Key Laboratory of Mathematical Modeling and High-Performance Scientific Computation, Xiamen University, Xiamen 361005, China
Department of Mechanical Engineering, CarnegieMellon University, Pittsburgh, PA 15213, USA
Show Author Information

Graphical Abstract

Abstract

Mesh-based image warping techniques typically represent image deformation using linear functions on triangular meshes or bilinear functions on rectangular meshes. This enables simple and efficient implementation, but in turn, restricts the representation capability of the deformation, often leading to unsatisfactory warping results. We present a novel, flexible polygonal finite element (poly-FEM) method for content-aware image warping. Image deformation is represented by high-order poly-FEMs on a content-aware polygonal mesh with a cell distribution adapted to saliency information in the source image. This allows highly adaptive meshes and smoother warping with fewer degrees of freedom, thus significantly extending the flexibility and capability of the warping representation. Benefiting from the continuous formulation of image deformation, our poly-FEM warping method is able to compute the optimal image deformation by minimizing existing or even newly designed warping energies consisting of penalty terms for specific transformations. We demonstrate the versatility of the proposed poly-FEM warping method in representing different deformations and its superiority by comparing it to other existing state-of-the-art methods.

Keywords

image warpingfinite element method (FEM)polygonal elementmesh generation

References

[1]
Vaquero, D.; Turk, M.; Pulli, K.; Tico, M.; Gelfand, N. A survey of image retargeting techniques. In: Proceedings of the SPIE 7798, Applications of Digital Image Processing XXXIII, 779814, 2010.
Crossref
[2]
Kiess, J.; Kopf, S.; Guthier, B.; Effelsberg, W. A survey on content-aware image and video retargeting. ACM Transactions on Multimedia Computing, Communications, and Applications Vol. 14, No. 3, Article No. 76, 2018.
Crossref Google Scholar
[3]
Suh, B.; Ling, H.; Bederson, B. B.; Jacobs. D. W. Automatic thumbnail cropping and its effectiveness. In: Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology, 95–104, 2003.
Crossref
[4]
Nishiyama, M.; Okabe, T.; Sato, Y.; Sato, I. Sensation-based photo cropping. In: Proceedings of the 17th ACM International Conference on Multimedia, 669–672, 2009.
Crossref
[5]
Setlur, V.; Takagi, S.; Raskar, R.; Gleicher, M.; Gooch, B. Automatic image retargeting. In: Proceedings of the 4th International Conference on Mobile and Ubiquitous Multimedia, 59–68, 2005.
Crossref
[6]
Avidan, S.; Shamir, A. Seam carving for content-aware image resizing. ACM Transactions on Graphics Vol. 26, No. 3, 10–es, 2007.
Crossref Google Scholar
[7]
Rubinstein, M.; Shamir, A.; Avidan, S. Improved seam carving for video retargeting. ACM Transactions on Graphics Vol. 27, No. 3, 1–9, 2008.
Crossref Google Scholar
[8]
Rubinstein, M.; Shamir, A.; Avidan, S. Multi-operator media retargeting. ACM Transactions on Graphics Vo. 28, No. 3 Article No. 23, 2009.
Crossref Google Scholar
[9]
Barnes, C.; Shechtman, E.; Finkelstein, A.; Goldman, D. B. PatchMatch: A randomized correspondence algorithm for structural image editing. In: Proceedings of the ACM SIGGRAPH 2009 papers, Article No. 24, 2009.
Crossref
[10]
Pritch, Y.; Kav-Venaki, E.; Peleg, S. Shift-map image editing. In: Proceedings of the IEEE 12th International Conference on Computer Vision, 151–158, 2009.
Crossref
[11]
Guo, Y. W.; Liu, F.; Shi, J.; Zhou, Z. H.; Gleicher, M. Image retargeting using mesh parametrization. IEEE Transactions on Multimedia Vol. 11, No. 5, 856–867, 2009.
Crossref Google Scholar
[12]
Lau, C. P.; Yung, C. P.; Lui, L. M. Image retargeting via Beltrami representation. IEEE Transactions on Image Processing Vol. 27, No. 12, 5787–5801, 2018.
Crossref Google Scholar
[13]
Wolf, L.; Guttmann, M.; Cohen-Or, D. Non-homogeneous content-driven video-retargeting. In: Proceedings of the IEEE 11th International Conference on Computer Vision, 1–6, 2007.
Crossref
[14]
Wang, Y.-S.; Tai, C.-L.; Sorkine, O.; Lee, T.-Y. Optimized scale-and-stretch for image resizing. ACM Transactions on Graphics Vol. 27, No. 5, Article No. 118, 2008.
Crossref Google Scholar
[15]
Karni, Z.; Freedman, D.; Gotsman, C. Energy-based image deformation. Computer Graphics Forum Vol. 28, 1257–1268, 2009.
Crossref Google Scholar
[16]
Krähenbühl, P.; Lang, M.; Hornung, A.; Gross, M. A system for retargeting of streaming video. ACM Transactions on Graphics Vol. 28, No. 5, 1–10, 2009.
Crossref Google Scholar
[17]
Shi, M.; Yang, L.; Peng, G.; Xu, D. A content-aware image resizing method with prominent object size adjusted. In: Proceedings of the 17th ACM Symposium on Virtual Reality Software and Technology, 175–176, 2010.
Crossref
[18]
Chen, R. J.; Freedman, D.; Karni, Z.; Gotsman, C.; Liu, L. G. Content-aware image resizing by quadratic programming. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, 1–8, 2010.
Crossref
[19]
Cho, D.; Park, J.; Oh, T. H.; Tai, Y. W.; Kweon, I. S. Weakly- and self-supervised learning for content-aware deep image retargeting. In: Proceedings of the IEEE International Conference on Computer Vision, 4568–4577, 2017.
Crossref
[20]
Tan, W. M.; Yan, B.; Lin, C. M.; Niu, X. J. Cycle-IR: Deep cyclic image retargeting. IEEE Transactions on Multimedia Vol. 22, No. 7, 1730–1743, 2020.
Crossref Google Scholar
[21]
Zhou, Y.; Chen, Z. B.; Li, W. P. Weakly supervised reinforced multi-operator image retargeting. IEEE Transactions on Circuits and Systems for Video Technology Vol. 31, No. 1, 126–139, 2021.
Crossref Google Scholar
[22]
Danon, D.; Arar, M.; Cohen-Or, D.; Shamir, A. Image resizing by reconstruction from deep features. Computational Visual Media Vol. 7, No. 4, 453–466, 2021.
Crossref Google Scholar
[23]
Hu, S. M.; Li, C. F.; Zhang, H. Actual morphing: A physics-based approach to blending. In: Proceedings of the 9th ACM Symposium on Solid Modeling and Applications, 309–314, 2004.
[24]
Yan, H.-B.; Hu, S.-M.; Martin, R. Morphing based on strain field interpolation. Computer Animation and Virtual Worlds Vol. 15, Nos. 3–4, 443–452, 2004.
Crossref Google Scholar
[25]
Bajaj, C.; Zhang, Y.; Xu, G. Physically-based surface texture synthesis using a coupled finite element system. In: Advances in Geometric Modeling and Processing. Lecture Notes in Computer Science, Vol. 4975. Chen, F.; Jüttler, B. Eds. Springer Berlin Heidelberg, 344–357, 2008.
[26]
Jacobson, A.; Tosun, E.; Sorkine, O.; Zorin, D. Mixed finite elements for variational surface modeling. Computer Graphics Forum Vol. 29, No. 5, 1565–1574, 2010.
Crossref Google Scholar
[27]
Zhang, Y. J. Geometric Modeling and Mesh Generation from Scanned Images. New York: Chapman and Hall/CRC, 2016.
Crossref
[28]
Gee, J. C.; Haynor, D. R.; Martin, R. M. D.; Bajcsy, R. K. Finite element approach to warping of brain images. In: Proceedings of the SPIE 2167, Medical Imaging 1994: Image Processing, 327–337, 1994.
[29]
Kaufmann, P.; Wang, O.; Sorkine-Hornung, A.; Sorkine-Hornung, O.; Smolic, A.; Gross, M. Finite element image warping. Computer Graphics Forum Vol. 32, No. 2pt1, 31–39, 2013.
Crossref Google Scholar
[30]
Wachspress, E. L. A Rational Finite Element Basis. Elsevier, 1975.
[31]
Meyer, M.; Barr, A.; Lee, H.; Desbrun, M. Generalized barycentric coordinates on irregular polygons. Journal of Graphics Tools Vol. 7, No. 1, 13–22, 2002.
Crossref Google Scholar
[32]
Hormann, K.; Floater, M. S. Mean value coordinates for arbitrary planar polygons. ACM Transactions on Graphics Vol. 25, No. 4, 1424–1441, 2006.
Crossref Google Scholar
[33]
Rand, A.; Gillette, A.; Bajaj, C. Quadratic serendipity finite elements on polygons using generalized barycentric coordinates. Mathematics of Computation Vol. 83, 2691–2716, 2014.
Crossref Google Scholar
[34]
Sukumar, N. Quadratic maximum-entropy serendipity shape functions for arbitrary planar polygons. Computer Methods in Applied Mechanics and Engineering Vol. 263, 27–41, 2013.
Crossref Google Scholar
[35]
Floater, M. S.; Lai, M. J. Polygonal spline spaces and the numerical solution of the Poisson equation. SIAM Journal on Numerical Analysis Vol. 54, No. 2, 797–824, 2016.
Crossref Google Scholar
[36]
Cao, J.; Xiao, Y. Y.; Chen, Z. G.; Wang, W. P.; Bajaj, C. Functional data approximation on bounded domains using polygonal finite elements. Computer Aided Geometric Design Vol. 63, 149–163, 2018.
Crossref Google Scholar
[37]
Engelke, U.; Wang, J. L.; Marendy, P. Perceptual relevance based image retargeting. IEEE Signal Processing Letters Vol. 22, No. 6, 705–708, 2015.
Crossref Google Scholar
[38]
Wang, J.; Jiang, H.; Yuan, Z.; Cheng, M.-M.; Hu, X.; Zheng, N. Salient object detection: A discriminative regional feature integration approach. International Journal of Computer Vision Vol. 123, No. 2, 251–268, 2017.
Crossref Google Scholar
[39]
Schneider, T. Theory and applications of bijective barycentric mappings. Ph.D. Thesis. Universita della Svizzera italiana, 2017.
Crossref
[40]
Sieger, D.; Alliez, P.; Botsch, M. Optimizing Voronoi Diagrams for polygonal finite element computations. In: Proceedings of the 19th International Meshing Roundtable, 335–350, 2010.
Crossref
[41]
Du, Q.; Faber, V.; Gunzburger, M. Centroidal voronoi tessellations: Applications and algorithms. SIAM Review Vol. 41, No. 4, 637–676, 1999.
Crossref Google Scholar
[42]
Gupta, S.; Mazumdar, S. G. Sobel edge detection algorithm. International Journal of Computer Science and Management Research Vol. 2, No. 2, 1578–1583, 2013.
Google Scholar
[43]
Laffont, P.-Y.; Jun, J. Y.; Wolf, C.; Tai, Y.-W.; Idrissi, K.; Drettakis, G.; Yoon, S.-e. Interactive content-aware zooming. In: Proceedings of the Graphics Interface, 79–87, 2010.
[44]
Zhang, G.-X.; Cheng, M.-M.; Hu, S.-M.; Martin, R. R. A shape-preserving approach to image resizing. Computer Graphics Forum Vol. 28, No. 7, 1897–1906, 2009.
Crossref Google Scholar
[45]
Tang, F.; Dong, W.; Meng, Y.; Ma, C.; Wu, F.; Li, X.; Lee, T.-Y. Image retargetability. IEEE Transactions on Multimedia Vol. 22, No. 3, 641–654, 2020.
Crossref Google Scholar
[46]
Rubinstein, M.; Gutierrez, D.; Sorkine, O.; Shamir, A. A comparative study of image retargeting. ACM Transactions on Graphics Vol. 29. No. 6, Article No. 160, 2010.
Crossref Google Scholar
[47]
Liu, Y. J.; Han, Y. H.; Ye, Z. P.; Lai, Y. K. Ranking-preserving cross-source learning for image retargeting quality assessment. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 42, No. 7, 1798–1805, 2020.
Crossref Google Scholar
[48]
Li, X. Y.; Ju, T.; Hu, S. M. Cubic mean value coordinates. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 126, 2013.
Crossref Google Scholar
Computational Visual Media
Volume 9 Issue 2,
June 2023
Pages 367-383
DOI: 10.1007/s41095-022-0283-7
Cite this article:
Cao J, Zhang X, Huang J, et al. Polygonal finite element-based content-aware image warping. Computational Visual Media, 2023, 9(2): 367-383. https://doi.org/10.1007/s41095-022-0283-7

8482

Views

100

Downloads

1

Crossref

1

Web of Science

1

Scopus

0

CSCD

Altmetrics
Received: 09 February 2022
Accepted: 08 March 2022
Published: 03 January 2023
© The Author(s) 2022.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
Return