Real-time distance field acceleration based free-viewpoint video synthesis for large sports fields

Yanran Dai; Jing Li; Yuqi Jiang; Haidong Qin; Bang Liang; Shikuan Hong; Haozhe Pan; Tao Yang

doi:10.1007/s41095-022-0323-3

AI Chat Paper

Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.

Chat more with AI

| Sign up

Browse by Subject

Search for peer-reviewed journals with full access.

Journals A - Z

About Us

Discover the SciOpen Platform and Achieve Your Research Goals with Ease.

About Us

Publish with Us

Support

Search articles, authors, keywords, DOl and etc.

Published Date

Reset Search

{{expandStatus?'Exit ':''}}Advanced Search

Journals A - Z

About Us

Publish with Us

Support

PDF (10.9 MB)

Cite

EndNote(RIS) BibTeX

Collect

Submit Manuscript

AI Chat Paper

Show Outline

Outline

Show full outline

Hide outline

Outline

Show full outline

Hide outline

Research Article | Open Access

Real-time distance field acceleration based free-viewpoint video synthesis for large sports fields

Yanran Dai^¹, Jing Li^¹(

), Yuqi Jiang^¹, Haidong Qin^², Bang Liang^², Shikuan Hong^¹, Haozhe Pan^¹, Tao Yang^²

1School of Telecommunications Engineering, XidianUniversity, Xi’an 710071, China

2National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, SAIIP, the School of Computer Science, Northwestern Polytechnical University, Xi’an 710129, China

Show Author Information

Graphical Abstract

Abstract

Free-viewpoint video allows the user to view objects from any virtual perspective, creating an immersive visual experience. This technology enhances the interactivity and freedom of multimedia performances. However, many free-viewpoint video synthesis methods hardly satisfy the requirement to work in real time with high precision, particularly for sports fields having large areas and numerous moving objects. To address these issues, we propose a free-viewpoint video synthesis method based on distance field acceleration. The central idea is to fuse multi-view distance field information and use it to adjust the search step size adaptively. Adaptive step size search is used in two ways: for fast estimation of multi-object three-dimensional surfaces, and synthetic view rendering based on global occlusion judgement. We have implemented our ideas using parallel computing for interactive display, using CUDA and OpenGL frameworks, and have used real-world and simulated experimental datasets for evaluation. The results show that the proposed method can render free-viewpoint videos with multiple objects on large sports fields at 25 fps. Furthermore, the visual quality of our synthetic novel viewpoint images exceeds that of state-of-the-art neural-rendering-based methods.

Keywords

free-viewpoint video view synthesis camera array distance field sports video

Electronic Supplementary Material

Video

41095_0323_ESM.mp4

References

[1]

Fukushima,

; Fujii,

; Ishibashi,

; Yendo,

; Tanimoto,

Real-time free viewpoint image rendering by using fast multi-pass dynamic programming. In: Proceedings of the 3DTV-Conference: The TrueVision - Capture, Transmission and Display of 3D Video, 1–4, 2010.

Crossref

[2]

Wang,

R. G.

; Luo,

J. J.

; Jiang,

X. B.

; Wang,

Z. Y.

; Wang,

W. M.

; Li,

; Gao,

Accelerating image-domain-warping virtual view synthesis on GPGPU. IEEE Transactions on Multimedia Vol. 19, No. 6, 1392–1400, 2017.

Crossref Google Scholar

[3]

Ceulemans,

; Lu,

S. P.

; Lafruit,

; Munteanu,

Robust multiview synthesis for wide-baseline camera arrays. IEEE Transactions on Multimedia Vol. 20, No. 9, 2235–2248, 2018.

Crossref Google Scholar

[4]

Cheung,

C. H.

; Sheng,

; Ngan,

K. N.

Motion compensated virtual view synthesis using novel particle cell. IEEE Transactions on Multimedia Vol. 23, 1908–1923, 2021.

Crossref Google Scholar

[5]

Nonaka,

; Sabirin,

; Chen,

; Sankoh,

; Naito,

Optimal billboard deformation via 3D voxel for free-viewpoint system. IEICE Transactions on Information and Systems Vol. E101.D, No. 9, 2381–2391, 2018.

Crossref Google Scholar

[6]

Sabirin,

; Yao,

; Nonaka,

; Sankoh,

; Naito,

Toward real-time delivery of immersive sports content. IEEE MultiMedia Vol. 25, No. 2, 61–70, 2018.

Crossref Google Scholar

[7]

Sankoh,

; Naito,

; Nonaka,

; Sabirin,

; Chen,

Robust billboard-based, free-viewpoint video synthesis algorithm to overcome occlusions under challenging outdoor sport scenes. In: Proceedings of the 26th ACM International Conference on Multimedia, 1724–1732, 2018.

Crossref

[8]

Yao,

; Nonaka,

; Sankoh,

; Naito,

Robust moving camera calibration for synthesizing free viewpoint soccer video. In: Proceedings of the IEEE International Conference on Image Processing, 1185–1189, 2016.

Crossref

[9]

Chen,

; Watanabe,

; Nonaka,

; Konno,

; Sankoh,

; Naito,

A robust billboard-based free-viewpoint video synthesizing algorithm for sports scenes. arXiv preprint arXiv:1908.02446, 2019.

Crossref Google Scholar

[10]

Yamada,

; Sankoh,

; Sugano,

; Naito,

Occlusion robust free-viewpoint video synthesis based on inter-camera/-frame interpolation. In: Proceedings of the IEEE International Conference on Image Processing, 2072–2076, 2013.

Crossref

[11]

Shin,

; Kasuya,

; Kitahara,

; Kameda,

; Ohta,

A comparison between two 3D free-viewpoint generation methods: Player-billboard and 3D reconstruction. In: Proceedings of the 3DTV-Conference: The True Vision -Capture, Transmission and Display of 3D Video, 1–4, 2010.

Crossref

[12]

Carballeira,

; Carmona,

; Díaz,

; Berjón,

; Corregidor,

; Cabrera,

; Morán,

; Doblado,

; Arnaldo,

; del Mar Martín,

; et al. FVV live: A real-time free-viewpoint video system with consumer electronics hardware. IEEE Transactions on Multimedia Vol. 24, 2378–2391, 2022.

Crossref Google Scholar

[13]

Hedman,

; Philip,

; Price,

; Frahm,

J. M.

; Drettakis,

; Brostow,

Deep blending for free-viewpoint image-based rendering. ACM Transactions on Graphics Vol. 37, No. 6, Article No. 257, 2018.

Crossref Google Scholar

[14]

Do,

; Bravo,

; Zinger,

; De With,

P. H. N.

GPU-accelerated real-time free-viewpoint DIBR for 3DTV. IEEE Transactions on Consumer Electronics Vol. 58, No. 2, 633–640, 2012.

Crossref Google Scholar

[15]

Gao,

X. S.

; Li,

K. Q.

; Chen,

W. Q.

; Yang,

Z. Y.

; Wei,

W. G.

; Cai,

Y. G.

Free viewpoint video synthesis based on DIBR. In: Proceedings of the IEEE Conference on Multimedia Information Processing and Retrieval, 275–278, 2020.

Crossref

[16]

Li,

; Zhu,

; Sun,

M. T.

Hole filling with multiple reference views in DIBR view synthesis. IEEE Transactions on Multimedia Vol. 20, No. 8, 1948–1959, 2018.

Crossref Google Scholar

[17]

Tian,

S. S.

; Zhang,

; Morin,

; Déforges,

A benchmark of DIBR synthesized view quality assessment metrics on a new database for immersive media applications. IEEE Transactions on Multimedia Vol. 21, No. 5, 1235–1247, 2019.

Crossref Google Scholar

[18]

Wang,

X. J.

; Shao,

; Jiang,

Q. P.

; Meng,

X. C.

; Ho,

Y. S.

Measuring coarse-to-fine texture and geometric distortions for quality assessment of DIBR-synthesized images. IEEE Transactions on Multimedia Vol. 23, 1173–1186, 2021.

Crossref Google Scholar

[19]

Jin,

; Wang,

A. H.

; Zhao,

; Lin,

C. Y.

; Zeng,

Region-aware 3-D warping for DIBR. IEEE Transactions on Multimedia Vol. 18, No. 6, 953–966, 2016.

Crossref Google Scholar

[20]

Liu,

Z. M.

; Jia,

; Yang,

; Luo,

P. Y.

; Guo,

; Tan,

M. K.

Deep view synthesis via self-consistent generative network. IEEE Transactions on Multimedia Vol. 24, 451–465, 2022.

Crossref Google Scholar

[21]

Zhou,

T. H.

; Tucker,

; Flynn,

; Fyffe,

; Snavely,

Stereo magnification: Learning view synthesis using multiplane images. ACM Transactions on Graphics Vol. 37, No. 4, Article No. 65, 2018.

Crossref Google Scholar

[22]

Wang,

Y. R.

; Huang,

Z. H.

; Zhu,

; Li,

; Cao,

; Yang,

R. G.

Interactive free-viewpoint video generation. Virtual Reality & Intelligent Hardware Vol. 2, No. 3, 247–260, 2020.

Crossref Google Scholar

[23]

Broxton,

; Flynn,

; Overbeck,

; Erickson,

; Hedman,

; Duvall,

; Dourgarian,

; Busch,

; Whalen,

; Debevec,

Immersive light field video with a layered mesh representation. ACM Transactions on Graphics Vol. 39, No. 4, Article No. 86, 2020.

Crossref Google Scholar

[24]

Broxton,

; Busch,

; Dourgarian,

; DuVall,

; Erickson,

; Evangelakos,

; Flynn,

; Hedman,

; Overbeck,

; Whalen,

; et al. DeepView immersive light field video. In: Proceedings of the ACM SIGGRAPH Immersive Pavilion, Article No. 15, 2020.

Crossref

[25]

Flynn,

; Broxton,

; Debevec,

; DuVall,

; Fyffe,

; Overbeck,

; Snavely,

; Tucker,

DeepView: View synthesis with learned gradient descent. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2362–2371, 2019.

Crossref

[26]

Penner,

; Zhang,

Soft 3D reconstruction for view synthesis. ACM Transactions on Graphics Vol. 36, No. 6, Article No. 235, 2017.

Crossref Google Scholar

[27]

Dou,

M. S.

; Khamis,

; Degtyarev,

; Davidson,

; Fanello,

S. R.

; Kowdle,

; Escolano,

S. O.

; Rhemann,

; Kim,

; Taylor,

; et al. Fusion4D: Real-time performance capture of challenging scenes. ACM Transactions on Graphics Vol. 35, No. 4, Article No. 114, 2016.

Crossref Google Scholar

[28]

Wei,

D. X.

; Xu,

X. W.

; Shen,

H. B.

; Huang,

K. J.

GAC-GAN: A general method for appearance-controllable human video motion transfer. IEEE Transactions on Multimedia Vol. 23, 2457–2470, 2021.

Crossref Google Scholar

[29]

Collet,

; Chuang,

; Sweeney,

; Gillett,

; Evseev,

; Calabrese,

; Hoppe,

; Kirk,

; Sullivan,

High-quality streamable free-viewpoint video. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 69, 2015.

Crossref Google Scholar

[30]

Huang,

; Li,

T. Y.

; Chen,

W. K.

; Zhao,

Y. J.

; Xing,

; LeGendre,

; Luo,

L. J.

; Ma,

C. Y.

; Li,

Deep volumetric video from very sparse multi-view performance capture. In: Computer Vision – ECCV 2018. Lecture Notes in Computer Science, Vol. 11220. Ferrari,

; Hebert,

; Sminchisescu,

; Weiss,

Eds. Springer Cham, 351–369, 2018.

Crossref

[31]

Natsume,

; Saito,

; Huang,

; Chen,

W. K.

; Ma,

C. Y.

; Li,

; Morishima,

SiCloPe: Silhouette-based clothed people. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4475–4485, 2019.

Crossref

[32]

Leroy,

; Franco,

J. S.

; Boyer,

Volume sweeping: Learning photoconsistency for multi-view shape reconstruction. International Journal of Computer Vision Vol. 129, No. 2, 284–299, 2021.

Crossref Google Scholar

[33]

Zheng,

Z. R.

; Yu,

; Liu,

Y. B.

; Dai,

Q. H.

PaMIR: Parametric model-conditioned implicit representation for image-based human reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 44, No. 6, 3170–3184, 2022.

Crossref Google Scholar

[34]

Meerits,

; Thomas,

; Nozick,

; Saito,

FusionMLS: Highly dynamic 3D reconstruction with consumer-grade RGB-D cameras. Computational Visual Media Vol. 4, No. 4, 287–303, 2018.

Crossref Google Scholar

[35]

Li,

J. W.

; Gao,

; Wu,

Y. H.

; Liu,

Y. D.

; Shen,

Y. F.

High-quality indoor scene 3D reconstruction with RGB-D cameras: A brief review. Computational Visual Media Vol. 8, No. 3, 369–393, 2022.

Crossref Google Scholar

[36]

Nonaka,

; Watanabe,

; Chen,

; Sabirin,

; Naito,

Fast plane-based free-viewpoint synthesis for real-time live streaming. In: Proceedings of the IEEE Visual Communications and Image Processing, 1–4, 2018.

Crossref

[37]

Yusuke,

; Takahashi,

; Fujii,

Free viewpoint video generation system using visual hull. In: Proceedings of the International Workshop on Advanced Image Technology, 1–4, 2018.

Crossref

[38]

Chen,

; Watanabe,

; Nonaka,

; Konno,

; Sankoh,

; Naito,

Fast free-viewpoint video synthesis algorithm for sports scenes. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 3209–3215, 2019.

Crossref

[39]

Watanabe,

; Tanaka,

Free viewpoint video synthesis on human action using shape from silhouette method. In: Proceedings of the SICE Annual Conference, 2748–2751, 2010.

[40]

Dellaert,

; Yen-Chen,

Neural volume rendering: NeRF and beyond. arXiv preprint arXiv:2101.05204, 2020.

Google Scholar

[41]

Mildenhall,

; Srinivasan,

P. P.

; Tancik,

; Barron,

J. T.

; Ramamoorthi,

; Ng,

NeRF: Representing scenes as neural radiance fields for view synthesis. In: Computer Vision – ECCV 2020. Lecture Notes in Computer Science, Vol. 12346. Vedaldi,

; Bischof,

; Brox,

; Frahm,

J. M.

Eds. Springer Cham, 405–421, 2020.

Crossref

[42]

Lombardi,

; Simon,

; Schwartz,

; Zollhoefer,

; Sheikh,

; Saragih,

Mixture of volumetric primitives for efficient neural rendering. ACM Transactions on Graphics Vol. 40, No. 4, Article No. 59, 2021.

Crossref Google Scholar

[43]

Yu,

; Li,

R. L.

; Tancik,

; Li,

; Ng,

; Kanazawa,

PlenOctrees for real-time rendering of neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 5732–5741, 2021.

Crossref

[44]

Fridovich-Keil,

; Yu,

; Tancik,

; Chen,

Q. H.

; Recht,

; Kanazawa,

Plenoxels: Radiance fields without neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5491–5500, 2022.

Crossref

[45]

Müller,

; Evans,

; Schied,

; Keller,

Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics Vol. 41, No. 4, Article No. 102, 2022.

Crossref Google Scholar

[46]

Peng,

S. D.

; Zhang,

Y. Q.

; Xu,

Y. H.

; Wang,

Q. Q.

; Shuai,

; Bao,

H. J.

; Zhou,

X. W.

Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9050–9059, 2021.

Crossref

[47]

Deng,

K. L.

; Liu,

; Zhu,

J. Y.

; Ramanan,

Depth-supervised NeRF: Fewer views and faster training for free. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12872–12881, 2022.

Crossref

[48]

Martin-Brualla,

; Radwan,

; Sajjadi,

M. S. M.

; Barron,

J. T.

; Dosovitskiy,

; Duckworth,

NeRF in the wild: Neural radiance fields for unconstrained photo collections. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7206–7215, 2021.

Crossref

[49]

Abdel-Aziz,

Y. I.

; Karara,

H. M.

; Hauck,

Direct linear transformation from comparator coordinates into object space coordinates in close-range photogrammetry. Photogrammetric Engineering & Remote Sensing Vol. 81, No. 2, 103–107, 2015.

Crossref Google Scholar

[50]

Chen,

L. C.

; Zhu,

Y. K.

; Papandreou,

; Schroff,

; Adam,

Encoder–decoder with atrous separable convolution for semantic image segmentation. In: Computer Vision – ECCV 2018. Lecture Notes in Computer Science, Vol. 11211. Ferrari,

; Hebert,

; Sminchisescu,

; Weiss,

Eds. Springer Cham, 833–851, 2018.

Crossref

[51]

Shah,

; Dey,

; Lovett,

; Kapoor,

AirSim: High-fidelity visual and physical simulation for autonomous vehicles. In: Field and Service Robotics. Springer Proceedings in Advanced Robotics, Vol. 5. Hutter,

; Siegwart,

Eds. Springer Cham, 621–635, 2018.

Crossref

Computational Visual Media

Volume 10 Issue 2,
April 2024

Pages 331-353

DOI: 10.1007/s41095-022-0323-3

Cite this article:

Dai Y, Li J, Jiang Y, et al. Real-time distance field acceleration based free-viewpoint video synthesis for large sports fields. Computational Visual Media, 2024, 10(2): 331-353. https://doi.org/10.1007/s41095-022-0323-3

452

Views

Downloads

Crossref

Web of Science

Scopus

CSCD

Google Scholar
Citation

Altmetrics

Received: 13 April 2022

Accepted: 26 October 2022

Published: 03 January 2024

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.