Home Computational Visual Media Article
PDF (11 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript
Show Outline
Figures (10)

Show 1 more figures Hide 1 figures
Tables (3)
Table 1
Table 2
Table 3
Research Article | Open Access

Joint head pose and facial landmark regression from depth images

Jie Wang1, Juyong Zhang1(), Changwei Luo1, Falai Chen1
University of Science and Technology of China, Hefei, Anhui, 230026, China.
Show Author Information

Abstract

This paper presents a joint head pose and facial landmark regression method with input from depth images for realtime application. Our main contributions are: firstly, a joint optimization method to estimate head pose and facial landmarks, i.e., the pose regression result provides supervised initialization for cascaded facial landmark regression, while the regression result for the facial landmarks can also help to further refine the head pose at each stage. Secondly, we classify the head pose space into 9 sub-spaces, and then use a cascaded random forest with a global shape constraint for training facial landmarks in each specific space. This classification-guided method can effectively handle the problem of large pose changes and occlusion. Lastly, we have built a 3D face database containing 73 subjects, each with 14 expressions in various head poses. Experiments on challenging databases show our method achieves state-of-the-art performance on both head pose estimation and facial landmark regression.

Keywords

head posefacial landmarksdepth images

References

[1]
C. Cao,; Y. Weng,; S. Lin,; K. Zhou, 3D shape regression for real-time facial animation. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 41, 2013.
Crossref Google Scholar
[2]
C. Cao,; Q. Hou,; K. Zhou, Displaced dynamic expression regression for real-time facial tracking and animation. ACM Transactions on Graphics Vol. 33, No. 4, Article No. 43, 2014.
Crossref Google Scholar
[3]
M. D. Breitenstein,; D. Kuettel,; T. Weise,; L. van Gool,; H. Pfister, Real-time face pose estimation from single range images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1-8, 2008.
Crossref
[4]
G. P. Meyer,; S. Gupta,; I. Frosio,; D. Reddy,; J. Kautz, Robust model-based 3D head pose estimation. In: Proceedings of the IEEE International Conference on Computer Vision, 3649-3657, 2015.
Crossref
[5]
P. Padeleris,; X. Zabulis,; A. A. Argyros, Head pose estimation on depth based on particle swarm optimation. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 42-49, 2012.
Crossref
[6]
E. Seeman,; K. Nickel,; R. Stiefelhagen, Head pose estimation using stereo vision for human-robot interaction. In: Proceedings of the 6th IEEE International Conference on Automatic Face and Gesture Recognition, 626-631, 2004.
[7]
S. Tulyakov,; R. L. Vieriu,; S. Semeniuta,; N. Sebe, Robust real-time extreme head pose estimation. In: Proceedings of the 22nd International Conference on Pattern Recognition, 2263-2268, 2014.
Crossref
[8]
X. P. Burgos-Artizzu,; P. Perona,; P. Dollar, Robust face landmark estimation under occlusion. In: Proceedings of the IEEE International Conference on Computer Vision, 151-1520, 2013.
Crossref
[9]
X. Cao,; Y. Wei,; F. Wei,; J. Sun, Face alignment by explicit shape regression. International Journal of Computer Vision Vol. 107, No. 2, 177-190, 2014.
Crossref Google Scholar
[10]
M. Dantone,; J. Gall,; G. Fanelli,; L. van Gool, Real-time facial feature detection using conditional regression forests. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2578-2585, 2012.
Crossref
[11]
Z. Zhang,; W. Zhang,; J. Liu,; X. Tang, Multiview facial landmark localization in RGB-D images via hierarchical regression with binary patterns. IEEE Transactions on Circuits and Systems for Video Technology Vol. 24, No. 9, 1475-1485, 2014.
Crossref Google Scholar
[12]
Z. Zhu,; R. R. Martin,; R. Pepperell,; A. Burleigh, 3D modeling and motion parallax for improved videoconferencing. Computational Visual Media Vol. 2, No. 2, 131-142, 2016.
Crossref Google Scholar
[13]
P. Dollár,; P. Welinder,; P. Perona, Cascaded pose regression. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1078-1085, 2010.
Crossref
[14]
X. Sun,; Y. Wei,; S. Liang,; X. Tang,; J. Sun, Cascaded hand pose regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 824-832, 2015.
Crossref
[15]
D. Chen,; S. Ren,; Y. Wei,; X. Cao,; J. Sun, Joint cascade face detection and alignment. In: Computer Vision-ECCV 2014. D. Fleet,; T. Pajdla,; B. Schiele,; T. Tuytelaars, Eds. Springer International Publishing Switzerland, 109-122, 2014.
[16]
D. Lee,; H. Park,; C. D. Yoo, Face alignment using cascade Gaussian process regression trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4204-4212, 2015.
Crossref
[17]
G. Tzimiropoulos, Project-out cascaded regression with an application to face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3659-3667, 2015.
Crossref
[18]
S. Ren,; X. Cao,; Y. Wei,; J. Sun, Face alignment at 3000 fps via regression local binary features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1685-1692, 2014.
Crossref
[19]
T. Baltrušaitis,; P. Robinson,; L. P. Morency, 3D constrained local model for rigid and non-rigid facial tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2610-2617, 2012.
Crossref
[20]
D. J. Tan,; F. Tombari,; N. Navab, A combined generalized and subject-specific 3D head pose estimation. In: Proceedings of the International Conference on 3D Vision, 500-508, 2015.
Crossref
[21]
E. Borovikov, Human head pose estimation by facial features location. arXiv preprint arXiv:1510.02774, 2015.
Google Scholar
[22]
G. Fanelli,; M. Dantone,; J. Gall,; A. Fossati,; L. van Gool, Random forests for real time 3D face analysis. International Journal of Computer Vision Vol. 101, No. 3, 437-458, 2013.
Crossref Google Scholar
[23]
G. Fanelli,; J. Gall,; L. van Gool, Real time head pose estimation with random regression forests. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 617-624, 2011.
Crossref
[24]
G. Fanelli,; T. Weise,; J. Gall,; L. van Gool, Real time head pose estimation from consumer depth cameras. In: Pattern Recognition. R. Mester,; M. Felsberg, Eds. Springer-Verlag Berlin Heidelberg, 101-110, 2011.
Crossref
[25]
C. Papazov,; T. K. Marks,; M. Jones, Real-time 3D head pose and facial landmark estimation from depth images using triangular surface patch features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4722-4730, 2015.
Crossref
[26]
T. F. Cootes,; G. J. Edwards,; C. J. Taylor, Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 23, No. 6, 681-685, 2001.
Crossref Google Scholar
[27]
D. Cristinacce,; T. Cootes, Boost regression active shape models. In: Proceedings of the British Machine Conference, 79.1-79.10, 2007.
Crossref
[28]
P. Sauer,; T. Cootes,; C. Taylor, Accurate regression procedures for active appearance models. In: Proceedings of the British Machine Vision Conference, 30.1-30.11, 2011.
Crossref
[29]
G. Tzimiropoulos,; M. Pantic, Optimization problems for fast AAM fitting in-the-wild. In: Proceedings of the IEEE International Conference on Computer Vision, 593-600, 2013.
Crossref
[30]
J. Xiao,; S. Baker,; I. Matthews,; T. Kanade, Real-time combined 2D+3D active appearance models. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 535-542, 2004.
[31]
M. C. Ruiz,; J. Illingworth, Automatic landmarking of faces in 3D-ALF3⁢D. In: Proceedings of the 5th International Conference on Visual Information Engineering, 41-46, 2008.
Crossref
[32]
S. Z. Gilani,; F. Shafait,; A. Mian, Shape-based automatic detection of a large number of 3D facial landmarks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4639-4648, 2015.
Crossref
[33]
A. Jourabloo,; X. Liu, Pose-invariant 3D face alignment. In: Proceedings of the IEEE International Conference on Computer Vision, 3694-3702, 2015.
Crossref
[34]
A. Jourabloo,; X. Liu, Large-pose face alignment via CNN-based dense 3D model fitting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4188-4196, 2016.
Crossref
[35]
L. Breiman, Random forests. Machine Learning Vol. 45, No. 1, 5-32, 2001.
Crossref Google Scholar
[36]
S. Schulter,; C. Leistner,; P. Wohlhart,; P. M. Roth,; H. Bischof, Alternating regression forests for object detection and pose estimation. In: Proceedings of the IEEE International Conference on Computer Vision, 417-424, 2013.
Crossref
[37]
S. Schulter,; P. Wohlhart,; C. Leistner,; A. Saffari,; P. M. Roth,; H. Bischof, Alternating decision forests. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 508-515, 2013.
Crossref
[38]
C. Wan,; A. Yao,; L. van Gool, Direction matters: Hand pose estimation from local surface normals. arXiv preprint arXiv:1604.02657, 2016.
Crossref Google Scholar
[39]
S. Ren,; X. Cao,; Y. Wei,; J. Sun, Global refinement of random forest. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 723-730, 2015.
[40]
G. Fanelli,; M. Dantone,; L. van Gool, Real time 3D face alignment with random forests-based active appearance models. In: Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 1-8, 2013.
Crossref
Computational Visual Media
Volume 3 Issue 3,
September 2017
Pages 229-241
DOI: 10.1007/s41095-017-0082-8
Cite this article:
Wang J, Zhang J, Luo C, et al. Joint head pose and facial landmark regression from depth images. Computational Visual Media, 2017, 3(3): 229-241. https://doi.org/10.1007/s41095-017-0082-8
Metrics & Citations  
Article History
Copyright
Rights and Permissions
Return

10.1007/s41095-017-0082-8.T1Parameters used for training

DatabaseTraining samplesFeatures in each non-leaf nodeBinary testsMaximum tree depthMinimum samples in each leaf nodeTrees
BIWI804152040142010
B3D(AC) 243583210401082010
BIWI + B3D(AC) 22352752040132010
3DFEP6743104040152010

10.1007/s41095-017-0082-8.T2Head pose estimation results. The first four rows give results using different databases and our method; the associated training parameters are given in Table 1. The next three rows show results for head pose by transforming the estimated pose from axis-angle representation to Euler angle form (A → E & N). We also compare results with different loss functions: entropy and information gain. The last four rows give head pose estimation results from other methods

DatabaseMethod ψx/yaw (1/( ∘)) ψy/pitch (1/( ∘)) ψz/roll (1/( ∘)) tx (mm) ty (mm) tz (mm)
BIWIOur method0.02370.02830.01621.561.561.40
B3D(AC) 2Our method0.01160.01930.00250.661.000.51
BIWI + B3D(AC) 2Our method0.02410.02940.00921.352.021.32
3DFEPOur method0.04190.04650.01633.043.062.42
BIWIE & C1.18 ∘1.40 ∘1.61 ∘1.651.431.31
BIWIA → E & N1.16 ∘1.30 ∘1.51 ∘1.911.471.55
BIWIE & N with InfoGain9.23 ∘12.21 ∘5.49 ∘15.887.846.61
BIWI[20] (generalized)1.9 ∘1.3 ∘1.0 ∘1.33.11.8
BIWI[7] (CT)4.7 ∘7.7 ∘5.3 ∘—
BIWI[24] (DRRF)8.9 ∘8.5 ∘7.9 ∘14.6
BIWI[25] (TSP)2.5 ∘1.8 ∘2.9 ∘6.8

10.1007/s41095-017-0082-8.T3Estimation errors for facial landmark localization for B3D(AC) 2 and 3DFEP databases. M-initial: cascaded facial landmark localization with mean shape as initialization. V-based: cascaded facial landmark localization with position aware shape as initialization　(Unit: mm)

LandmarkB3D(AC) 23DFEPLandmarkB3D(AC) 23DFEP
FanelliM-initialV-basedV-basedFanelliM-initialV-basedV-based
LefEyeU—3.731.105.69RigNoseUC—3.871.155.17
LefEyeL4.43.731.205.94RigNoseU—3.711.075.25
LefEyeD—3.951.105.40NoseTip—3.811.146.11
LefEyeR3.43.671.096.44OutMouthL5.33.821.085.87
RigEyeU—3.751.355.31OutMouthUL—3.751.094.91
RigEyeL3.83.891.255.86OutMouthU3.53.741.215.50
RigEyeD—3.841.085.47OutMouthUR—3.651.165.73
RigEyeR4.83.621.304.96OutMouthR5.33.801.616.20
LefNoseU—3.691.195.40OutMouthDR—3.691.365.52
LefNoseUC—3.791.085.93OutMouthD6.23.801.226.25
LefNoseDC—3.721.085.85OutMouthDL—3.691.095.17
LefNoseD3.33.571.564.79InnMouthL—3.841.374.57
CenNoseL—3.691.386.02InnMouthU—3.941.494.47
CenNoseC—3.791.084.95InnMouthR—3.761.114.75
CenNoseR—3.721.186.04InnMouthD—3.841.104.59
RigNoseD2.93.601.486.04Chin—3.581.226.62
RigNoseDC—3.661.545.52—————