Comfort-driven disparity adjustment for stereoscopic video

Miao Wang; Xi-Jin Zhang; Jun-Bang Liang; Song-Hai Zhang; Ralph R. Martin

doi:10.1007/s41095-016-0037-5

AI Chat Paper

Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.

Chat more with AI

| Sign up

Browse by Subject

Search for peer-reviewed journals with full access.

Journals A - Z

About Us

Discover the SciOpen Platform and Achieve Your Research Goals with Ease.

About Us

Publish with Us

Support

Search articles, authors, keywords, DOl and etc.

Published Date

Reset Search

{{expandStatus?'Exit ':''}}Advanced Search

Journals A - Z

About Us

Publish with Us

Support

PDF (32.3 MB)

Cite

EndNote(RIS) BibTeX

Collect

Submit Manuscript

AI Chat Paper

Show Outline

Outline

Show full outline

Hide outline

Outline

Show full outline

Hide outline

Research Article | Open Access

Comfort-driven disparity adjustment for stereoscopic video

Miao Wang^¹(

), Xi-Jin Zhang^¹, Jun-Bang Liang^¹, Song-Hai Zhang^¹, Ralph R. Martin^²

1 Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China.

2 Cardiff University, Cardiff, CF243AA, UK.

Show Author Information

Abstract

Pixel disparity—the offset of corresponding pixels between left and right views—is a crucial parameter in stereoscopic three-dimensional (S3D) video, as it determines the depth perceived by the human visual system (HVS). Unsuitable pixel disparity distribution throughout an S3D video may lead to visual discomfort. We present a unified and extensible stereoscopic video disparity adjustment framework which improves the viewing experience for an S3D video by keeping the perceived 3D appearance as unchanged as possible while minimizing discomfort. We first analyse disparity and motion attributes of S3D video in general, then derive a wide-ranging visual discomfort metric from existing perceptual comfort models. An objective function based on this metric is used as the basis of a hierarchical optimisation method to find a disparity mapping function for each input video frame. Warping-based disparity manipulation is then applied to the input video to generate the output video, using the desired disparity mappings as constraints. Our comfort metric takes into account disparity range, motion, and stereoscopic window violation; the framework could easily be extended to use further visual comfort models. We demonstrate the power of our approach using both animated cartoons and real S3D videos.

Keywords

stereoscopic video editing video enhancement perceptual visual computing video manipulation

Electronic Supplementary Material

Download File(s)

41095_2016_37_MOESM1_ESM.pdf (423.5 KB)

References

[1]

Maimone,

; Wetzstein,

; Hirsch,

; Lanman,

; Raskar,

; Fuchs,

Focus 3D: Compressive accommodation display. ACM Transactions on Graphics Vol. 32, No. 5, Article No. 153, 2013.

Crossref Google Scholar

[2]

Wetzstein,

; Lanman,

; Heidrich,

; Raskar,

Layered 3D: Tomographic image synthesis for attenuation-based light field and high dynamic range displays. ACM Transactions on Graphics Vol. 30, No. 4, Article No. 95, 2011.

Crossref Google Scholar

[3]

Didyk,

; Ritschel,

; Eisemann,

; Myszkowski,

; Seidel,

H.-P.

A perceptual model for disparity. ACM Transactions on Graphics Vol. 30, No. 4, Article No. 96, 2011.

Crossref Google Scholar

[4]

Didyk,

; Ritschel,

; Eisemann,

; Myszkowski,

; Seidel,

H.-P.

; Matusik,

A luminance-contrast-aware disparity model and applications. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 184, 2012.

Crossref Google Scholar

[5]

Hoffman,

D. M.

; Girshick,

A. R.

; Akeley,

; Banks,

M. S.

Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. Journal of Vision Vol. 8, No. 3, 33, 2008.

Crossref Google Scholar

[6]

Howard,

I. P.

; Rogers,

B. J.

Perceiving in Depth, Vol. 2: Stereoscopic Vision. New York: Oxford University Press, 2012.

Crossref

[7]

Palmer,

S. E.

Vision Science: Photons to Phenomenology. Cambridge, MA, USA: MIT Press, 1999.

[8]

Mendiburu,

3D Movie Making: Stereoscopic Digital Cinema from Script to Screen. Oxon, UK: Focal Press, 2009.

[9]

Kellnhofer,

; Ritschel,

; Myszkowski,

; Seidel,

H.-P.

Optimizing disparity for motion in depth. Computer Graphics Forum Vol. 32, No. 4, 143-152, 2013.

Crossref Google Scholar

[10]

Lang,

; Hornung,

; Wang,

; Poulakos,

; Smolic,

; Gross,

Nonlinear disparity mapping for stereoscopic 3D. ACM Transactions on Graphics Vol. 29, No. 4, Article No. 75, 2010.

Crossref Google Scholar

[11]

Liu,

C.-W.

; Huang,

T.-H.

; Chang,

M.-H.

; Lee,

K.-Y.

; Liang,

C.-K.

; Chuang,

Y.-Y.

3D cinematography principles and their applications to stereoscopic media processing. In: Proceedings of the 19th ACM International Conference on Multimedia, 253-262, 2011.

Crossref

[12]

Shibata,

; Kim,

; Hoffman,

D. M.

; Banks,

M. S.

The zone of comfort: Predicting visual discomfort with stereo displays. Journal of Vision Vol. 11, No. 8, 11, 2011.

Crossref Google Scholar

[13]

Cho,

S.-H.

; Kang,

H.-B.

Subjective evaluation of visual discomfort caused from stereoscopic 3D video using perceptual importance map. In: Proceedings of IEEE Region 10 Conference, 1-6, 2012.

Crossref

[14]

Du,

S.-P.

; Masia,

; Hu,

S.-M.

; Gutierrez,

A metric of visual comfort for stereoscopic motion. ACM Transactions on Graphics Vol. 32, No. 6, Article No. 222, 2013.

Crossref Google Scholar

[15]

Jung,

Y. J.

; Lee,

S.-i.

; Sohn,

; Park,

H. W.

; Ro,

Y. M.

Visual comfort assessment metric based on salient object motion information in stereoscopic video. Journal of Electronic Imaging Vol. 21, No. 1, 011008, 2012.

Crossref Google Scholar

[16]

Kooi,

F. L.

; Toet,

Visual comfort of binocular and 3D displays. Displays Vol. 25, Nos. 2-3, 99-108, 2004.

Crossref Google Scholar

[17]

Mu,

T.-J.

; Sun,

J.-J.

; Martin,

R. R.

; Hu,

S.-M.

A response time model for abrupt changes in binocular disparity. The Visual Computer Vol. 31, No. 5, 675-687, 2015.

Crossref Google Scholar

[18]

Templin,

; Didyk,

; Myszkowski,

; Hefeeda,

M. M.

; Seidel,

H.-P.

; Matusik,

Modeling and optimizing eye vergence response to stereoscopic cuts. ACM Transactions on Graphics Vol. 33, No. 4, Article No. 145, 2014.

Crossref Google Scholar

[19]

Jin,

E. W.

; Miller,

M. E.

; Endrikhovski,

; Cerosaletti,

C. D.

Creating a comfortable stereoscopic viewing experience: Effects of viewing distance and field of view on fusional range. In: Proceedings of SPIE 5664, Stereoscopic Displays and Virtual Reality Systems XII, 10, 2005.

Crossref

[20]

Ukai,

; Howarth,

P. A.

Visual fatigue caused by viewing stereoscopic motion images: Background, theories, and observations. Displays Vol. 29, No. 2, 106-116, 2008.

Crossref Google Scholar

[21]

Zilly,

; Müller,

; Eisert,

; Kauff,

The stereoscopic analyzer—An image-based assistance tool for stereo shooting and 3D production. In: Proceedings of the 17th IEEE International Conference on Image Processing, 4029-4032, 2010.

Crossref

[22]

Masia,

; Wetzstein,

; Didyk,

; Gutierrez,

A survey on computational displays: Pushing the boundaries of optics, computation, and perception. Computers & Graphics Vol. 37, No. 8, 1012-1038, 2013.

Crossref Google Scholar

[23]

Lo,

W.-Y.

; van Baar,

; Knaus,

; Zwicker,

; Gross,

M. H.

Stereoscopic 3D copy & paste. ACM Transactions on Graphics Vol. 29, No. 6, Article No. 147, 2010.

Crossref Google Scholar

[24]

Tong,

R.-F.

; Zhang,

; Cheng,

K.-L.

StereoPasting: Interactive composition in stereoscopic images. IEEE Transactions on Visualization and Computer Graphic Vol. 19, No. 8, 1375-1385, 2013.

Crossref Google Scholar

[25]

Kim,

; Lee,

; Kang,

; Lee,

Stereoscopic 3D line drawing. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 57, 2013.

Crossref Google Scholar

[26]

Niu,

; Feng,

W.-C.

; Liu,

Enabling warping on stereoscopic images. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 183, 2012.

Crossref Google Scholar

[27]

Kim,

; Hornung,

; Heinzle,

; Matusik,

; Gross,

Multi-perspective stereoscopy from light fields. ACM Transactions on Graphics Vol. 30, No. 6, Article No. 190, 2011.

Crossref Google Scholar

[28]

Masia,

; Wetzstein,

; Aliaga,

; Raskar,

; Gutierrez,

Display adaptive 3D content remapping. Computers & Graphics Vol. 37, No. 8, 983-996, 2013.

Crossref Google Scholar

[29]

Koppal,

S. J.

; Zitnick,

C. L.

; Cohen,

; Kang,

S. B.

; Ressler,

; Colburn,

A viewer-centric editor for 3D movies. IEEE Computer Graphics and Applications Vol. 31, No. 1, 20-35, 2011.

Crossref Google Scholar

[30]

Oskam,

; Hornung,

; Bowles,

; Mitchell,

; Gross,

OSCAM-optimized stereoscopic camera control for interactive 3D. ACM Transactions on Graphics Vol. 30, No. 6, Article No. 189, 2011.

Crossref Google Scholar

[31]

Tseng,

K.-L.

; Huang,

W.-J.

; Luo,

A.-C.

; Huang,

W.-H.

; Yeh,

Y.-C.

; Chen,

W.-C.

Automatically optimizing stereo camera system based on 3D cinematography principles. In: Proceedings of 3DTV-Conference: The True Vision — Capture, Transmission and Display of 3D Video, 1- 4, 2012

[32]

Cheng,

M.-M.

; Mitra,

N. J.

; Huang,

; Torr,

P. H. S.

; Hu,

S.-M.

Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 37, No. 3, 569-582, 2015.

Crossref Google Scholar

[33]

Felzenszwalb,

P. F.

; Huttenlocher,

D. P.

Efficient graph-based image segmentation. International Journal of Computer Vision Vol. 59, No. 2, 167-181, 2004.

Crossref Google Scholar

[34]

Achanta,

; Shaji,

; Smith,

; Lucchi,

; Fua,

; Süsstrunk,

SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 34, No. 11, 2274-2282, 2012.

Crossref Google Scholar

[35]

Manning,

C. D.

; Schütze,

Foundations of Statistical Natural Language Processing. Cambridge, MA, USA: MIT Press, 1999.

[36]

Syswerda,

A study of reproduction in generational and steady state genetic algorithms. Foundation of Genetic Algorithms Vol. 2, 94-101, 1991.

Crossref Google Scholar

[37]

Syswerda,

Uniform crossover in genetic algorithms. In: Proceedings of the 3rd International Conference on Genetic Algorithms, 2-9, 1989.

[38]

Higashi,

; Iba,

Particle swarm optimization with Gaussian mutation. In: Proceedings of the IEEE Swarm Intelligence Symposium, 72-79, 2003.

[39]

Brox,

; Malik,

Large displacement optical flow: Descriptor matching in variational motion estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 33, No. 3, 500-513, 2011.

Crossref Google Scholar

Computational Visual Media

Volume 2 Issue 1,
March 2016

Pages 3-17

DOI: 10.1007/s41095-016-0037-5

Cite this article:

Wang M, Zhang X-J, Liang J-B, et al. Comfort-driven disparity adjustment for stereoscopic video. Computational Visual Media, 2016, 2(1): 3-17. https://doi.org/10.1007/s41095-016-0037-5

819

Views

Downloads

Crossref

N/A

Web of Science

Scopus

CSCD

Google Scholar
Citation

Altmetrics

Revised: 02 December 2015

Accepted: 09 December 2015

Published: 01 March 2016

This article is published with open access at Springerlink.com

The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.