A three-stage real-time detector for traffic signs in large panoramas

Yizhi Song; Ruochen Fan; Sharon Huang; Zhe Zhu; Ruofeng Tong

doi:10.1007/s41095-019-0152-1

| Sign up

PDF (21.6 MB)

Cite

EndNote(RIS) BibTeX

Collect

Submit Manuscript

Show Outline

Figures (5)

Fig. 1:

Fig. 2:

Fig. 3:

Fig. 4:

Fig. 5:

Tables (5)

Table 1:

Table 3:

Table 2:

Table 4:

Table 5:

Research Article | Open Access

A three-stage real-time detector for traffic signs in large panoramas

Yizhi Song^¹, Ruochen Fan^², Sharon Huang^³, Zhe Zhu^⁴, Ruofeng Tong^⁵()

1Department of Computer Science, Purdue University, 305 N. University Street, West Lafayette, IN 47907, USA.

2Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China.

3College of Information Sciences and Technology, Penn State University, University Park, PA 16802, USA.

4Department of Radiology, Duke University, Durham, NC 27705, USA.

5College of Computer Science and Technology,Zhejiang University, Hangzhou 310007, China.

Show Author Information

Abstract

Traffic sign detection is one of the key com-ponents in autonomous driving. Advanced autonomous vehicles armed with high quality sensors capture high definition images for further analysis. Detecting traffic signs, moving vehicles, and lanes is important for localization and decision making. Traffic signs, especially those that are far from the camera, are small, and so are challenging to traditional object detection methods. In this work, in order to reduce computational cost and improve detection performance, we split the large input images into small blocks and then recognize traffic signs in the blocks using another detection module. Therefore, this paper proposes a three-stage traffic sign detector, which connects a BlockNet with an RPN-RCNN detection network. BlockNet, which is composed of a set of CNN layers, is capable of performing block-level foreground detection, making inferences in less than 1 ms. Then, the RPN-RCNN two-stage detector is used to identify traffic sign objects in each block; it is trained on a derived dataset named TT100KPatch. Experiments show that our framework can achieve both state-of-the-art accuracy and recall; its fastest detection speed is 102 fps.

Keywords

traffic sign detection real time

References

[1]

Everingham,

; L.

van Gool,

; C. K. I.

Williams,

; J.

Winn,

; A.

Zisserman,

The Pascal visual object classes (VOC) challenge. International Journal of Computer Vision Vol. 88, No. 2, 303-338, 2010.

Crossref Google Scholar

[2]

Zhu,

; D.

Liang,

; S. H.

Zhang,

; X. L.

Huang,

; B. L.

Li,

; S. M.

Hu,

Traffic-sign detection and classification in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2110-2118, 2016.

[3]

Sermanet,

; D.

Eigen,

; X.

Zhang,

; M.

Mathieu,

; R.

Fergus,

; Y

LeCun,

. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229, 2013.

Google Scholar

[4]

Redmon,

; S.

Divvala,

; R.

Girshick,

; A.

Farhadi.

You only look once: Unified, real-time object detection. In: Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition, 779-788, 2016.

[5]

Liu,

; D.

Anguelov,

; D.

Erhan,

; C.

Szegedy,

; S.

Reed,

; C. Y.

Fu,

; A. C.

Berg,

SSD: Single shot MultiBox detector. In: Computer Vision-ECCV 2016. Lecture Notes in Computer Science, Vol. 9905. B.

Leibe,

; J.

Matas,

; N.

Sebe,

; M

Welling,

. Eds. Springer Cham, 21-37, 2016.

Crossref

[6]

Kong,

; F. C.

Sun,

; A. B.

Yao,

; H. P.

Liu,

; M.

Lu,

; Y. R.

Chen,

RON: Reverse connection with objectness prior networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5244-5252, 2017.

[7]

T. Y.

Lin,

; P.

Goyal,

; R.

Girshick,

; K. M.

He,

; P.

Dollar,

Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2018.

Crossref Google Scholar

[8]

Girshick,

; J.

Donahue,

; T.

Darrell,

; J.

Malik,

Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 580-587, 2014.

[9]

Girshick,

Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, 1440-1448, 2015.

[10]

Ren,

; K.

He,

; R.

Girshick,

; J.

Sun,

Faster R-CNN: Towards real-time object detection with region proposal networks In: Proceedings of the Advances in Neural Information Processing Systems 28, 91-99, 2015.

[11]

He,

; G.

Gkioxari,

; P.

Dollar,

; R.

Girshick,

Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, 2961-2969, 2017.

[12]

Everingham,

; L.

van Gool,

; C. K. I.

Williams,

; J.

Winn,

; A

Zisserman,

. The Pascal visual object classes challenge 2007 (voc2007) results. Available at http://www.pascal-network.org/challenges/VOC/ voc2007/workshop/index.html.

[13]

T. Y.

Lin,

; M.

Maire,

; S.

Belongie,

; J.

Hays,

; P.

Perona,

; D.

Ramanan,

; P.

Dollár,

; C. L.

Zitnick,

Microsoft COCO: Common objects in context. In: Computer Vision-ECCV 2014. Lecture Notes in Computer Science, Vol. 8693. D.

Fleet,

; T.

Pajdla,

; B.

Schiele,

; T

Tuytelaars,

. Eds. Springer Cham, 740-755, 2014.

Crossref

[14]

Houben,

; J.

Stallkamp,

; J.

Salmen,

; M.

Schlipsing,

; C.

Igel,

Detection of traffic signs in real-world images: The German traffic sign detection benchmark. In: Proceedings of the International Joint Conference on Neural Networks, 1-8, 2013.

[15]

Stallkamp,

; M.

Schlipsing,

; J.

Salmen,

; C.

Igel,

The German traffic sign recognition benchmark: A multi-class classification competition. In: Proceedings of the International Joint Conference on Neural Networks, 1453-1460, 2011.

[16]

Timofte,

; K.

Zimmermann,

; L.

van Gool,

Multi-view traffic sign detection, recognition, and 3D localisation. Machine Vision and Applications Vol. 25, No. 3, 633-647, 2014.

Crossref Google Scholar

[17]

V. B.

Hemadri,

; U. P.

Kulkarni,

Recognition of traffic sign based on support vector machine and creation of the Indian traffic sign recognition benchmark. In: Cognitive Computing and Information Processing. Communications in Computer and Information Science, Vol. 801. T.

Nagabhushan,

; V.

Aradhya,

; P.

Jagadeesh,

; S

Shukla,

. Eds. Springer Singapore, 227-238, 2018.

[18]

Larsson,

; M.

Felsberg,

Using Fourier descriptors and spatial models for traffic sign recognition. In: Image Analysis. Lecture Notes in Computer Science, Vol. 6688. A.

Heyden,

; F

Kahl,

. Eds. Springer Berlin Heidelberg, 238-249, 2011.

Crossref

[19]

Yang,

; H. L.

Luo,

; H. R.

Xu,

; F. C.

Wu,

Towards real-time traffic sign detection and classification. IEEE Transactions on Intelligent Transportation Systems Vol. 17, No. 7, 2022-2031, 2016.

Crossref Google Scholar

[20]

Meng,

; X.

Fan,

; X.

Chen,

; M.

Chen,

; Y.

Tong,

Detecting small signs from large images. In: Proceedings of the IEEE International Conference on Information Reuse and Integration, 217-224, 2017.

[21]

T. T.

Yang,

; X.

Long,

; A. K.

Sangaiah,

; Z. G.

Zheng,

; C.

Tong,

Deep detection network for real-life traffic sign in vehicular networks. Computer Networks Vol. 136, 95-104, 2018.

Crossref Google Scholar

[22]

Pon,

; O.

Adrienko,

; A.

Harakeh,

; S. L.

Waslander,

A hierarchical deep architecture and mini-batch selection method for joint traffic sign and light detection. In: Proceedings of the 15th Conference on Computer and Robot Vision, 102-109, 2018.

[23]

He,

; X.

Zhang,

; S.

Ren,

; J.

Sun,

Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770-778, 2015.

[24]

Yang,

; J.

Lu,

; D.

Batra,

; Parikh, D. A faster pytorch implementation of faster R-CNN 2017. Available at https://github.com/jwyang/faster-rcnn.pytorch.

[25]

Li,

; X.

Liang,

; Y.

Wei,

; T.

Xu,

; J.

Feng,

; S.

Yan,

Perceptual generative adversarial networks for small object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1222-1230, 2017.

[26]

Redmon,

; A

Farhadi,

. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.

Google Scholar

[27]

Lu,

; J.

Lu,

; S.

Zhang,

; P.

Hall,

Traffic signal detection and classification in street views using an attention model. Computational Visual Media Vol. 4, No. 3, 253-266, 2018.

Crossref Google Scholar

Computational Visual Media

Volume 5 Issue 4,
December 2019

Pages 403-416

DOI: 10.1007/s41095-019-0152-1

Cite this article:

Song Y, Fan R, Huang S, et al. A three-stage real-time detector for traffic signs in large panoramas. Computational Visual Media, 2019, 5(4): 403-416. https://doi.org/10.1007/s41095-019-0152-1

Return

10.1007/s41095-019-0152-1.T1Table 1:Detection results: performance of state-of-the-art works. acc: accuracy. rec: recall. Results of only using Faster R-CNN for the task are also shown. ${𝐎𝐮𝐫𝐬}^{†}$ : proposed framework with few block proposals for high speed. ${𝐎𝐮𝐫𝐬}^{‡}$ : proposed framework adjusted for best accuracy. Our proposed framework achieves the best recall and accuracy. The proposed BlockNet significantly boosts detection recall and accuracy with respect to a standard two-shot detection framework (Faster R-CNN)

Work	mAP	acc	rec
Zhu et al.	—	0.88	0.91
Li et al. [25]	—	0.89	0.91
Meng et al.	—	0.90	0.93
Pon et al.	0.31	0.68	0.44
Yang et al.	0.80	—	—
Faster R-CNN	0.38	0.64	0.37
${𝐎𝐮𝐫𝐬}^{†}$	0.52	0.69	0.65
${𝐎𝐮𝐫𝐬}^{‡}$	0.90	0.92	0.93

10.1007/s41095-019-0152-1.T3Table 3:Efficiency of state-of-the-art methods. Our proposed framework boosts the highest inference speed by 35%, while outperforming the work of Pon et al.

Work	mAP	inf speed (s)	fps
Li et al.	—	0.6	1.7
Yang et al.	0.80	0.1282	7.8
Pon et al.	0.31	0.015	66.7
YOLOv3 [26]	0.33	0.030	33.3
Faster R-CNN	0.38	0.022	45.5
Ours	0.52	0.0098	102.0

10.1007/s41095-019-0152-1.T2Table 2:Sensitivity to objects of different sizes. Small and medium objects are dominant in the dataset, verifying the difficulty of TT100K. gt: number of ground truth boxes whose area falls into the interval. rec: recall. The model can readily identify objects larger than $32^{2}$

Group	gt	ratio	rec
Small	11,525	52%	0.5392
Medium	9,983	45%	0.9386
Large	702	3%	0.9316

10.1007/s41095-019-0152-1.T4Table 4:Performance (using the fastest model trained) over 45 categories having more than 100 instances. mAP of different classes varies greatly due to varying numbers of instances and difficulty of extracting features. Classes with distinct geometrical characteristics (for instance, pne and pn), are easier to recognize, while triangular signs are harder to distinguish

Class	i2	i4	i5	il100	il60	il80	io	ip	p10	p11	p12	p19	p23	p26	p27
mAP	0.468	0.674	0.747	0.624	0.708	0.552	0.537	0.607	0.525	0.529	0.346	0.523	0.620	0.542	0.504
Class	p3	p5	p6	pg	ph4	ph4.5	ph5	pl100	pl120	pl20	pl30	pl40	pl5	pl50	pl60
mAP	0.489	0.716	0.318	0.544	0.332	0.573	0.363	0.682	0.581	0.358	0.390	0.560	0.642	0.419	0.465
Class	pl70	pl80	pm20	pm30	pm55	pn	pne	po	pr40	w13	w32	w55	w57	w59	wo
mAP	0.364	0.490	0.401	0.346	0.651	0.698	0.763	0.393	0.721	0.361	0.599	0.289	0.566	0.548	0.249

10.1007/s41095-019-0152-1.T5Table 5:Timing and performance. pth/img: average number of patches produced by an image. Numbers of proposals before and after NMS: PRE-NMS, POST-NMS. All inputs have a batch size of 24. The last three columns are inference time (in second) for BlockNet, the two-shot network and the whole pipeline = pth/img $\times$ inf2/img $+$ inf1/img. Decreasing the number of proposals after NMS and zooming out the blocks result in a loss in performance, though earning more time. Increasing the sensitivity of BlockNet leads to a growth in the total number of blocks selected, eventually causing the whole network to slow down

No.	pth/img	scale	PRE-NMS	POST-NMS	mAP	acc	rec	inf1/img	inf2/img	time/img
test1	4.44	600	6,000	300	0.7707	0.9211	0.8136	0.00090	0.02176	0.09749
test2	6.17	600	1,000	6	0.7437	0.9385	0.7715	0.00091	0.01876	0.11667
test3	4.44	256	1,000	6	0.6770	0.8658	0.7005	0.00091	0.00406	0.01894
test4	6.17	128	1,000	6	0.5141	0.6788	0.6523	0.00089	0.00142	0.00963
test5	6.29	128	1,000	6	0.5194	0.6908	0.6532	0.00089	0.00142	0.00980
test6	5.64	800	6,000	300	0.8989	0.9204	0.9327	0.00112	0.03541	0.20083