On the role of geometry in geo-localization

Moti Kadosh; Yael Moses; Ariel Shamir

doi:10.1007/s41095-020-0196-2

| Sign up

PDF (10.6 MB)

Cite

EndNote(RIS) BibTeX

Collect

Submit Manuscript

Show Outline

Figures (5)

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Tables (4)

Table 1

Table 2

Table 3

Table 4

Research Article | Open Access

On the role of geometry in geo-localization

Moti Kadosh^¹(), Yael Moses^², Ariel Shamir^²

1Department of Electrical Engineering, Tel-Aviv University, Tel-Aviv, Israel

2The Efi Arazi School of Computer Science, The Interdisciplinary Center Herzliya, Herzliya, Israel

Show Author Information

Abstract

Consider the geo-localization task of finding the pose of a camera in a large 3D scene from a single image. Most existing CNN-based methods use as input textured images. We aim to experimentally explore whether texture and correlation between nearby images are necessary in a CNN-based solution for the geo-localization task. To do so, we consider lean images, textureless projections of a simple 3D model of a city. They only contain information related to the geometry of the scene viewed (edges, faces, and relative depth). The main contributions of this paper are: (i) todemonstrate the ability of CNNs to recover camera pose using lean images; and (ii) to provide insight into the role of geometry in the CNN learning process.

Keywords

geo-localization geometry CNN-based solutions synthetic lean images

References

[1]

Se,

; Lowe,

; Little,

Mobile robot localization and mapping with uncertainty using scale-invariant visual landmarks. The International Journal of Robotics Research Vol. 21, No. 8, 735-758, 2002.

Crossref Google Scholar

[2]

Lowe,

D. G.

Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision Vol. 60, No. 2, 91-110, 2004.

Crossref Google Scholar

[3]

Li,

Y. P.

; Snavely,

; Huttenlocher,

D. P.

Location recognition using prioritized feature matching. In: Computer Vision - ECCV 2010. Lecture Notes in Computer Science, Vol. 6312. Daniilidis,

; Maragos,

; Paragios,

Eds. Springer Berlin Heidelberg, 791-804, 2010.

[4]

Ramalingam,

; Bouaziz,

; Sturm,

Pose estimation using both points and lines for geo-localization. In: Proceedings of the IEEE International Conference on Robotics and Automation, 4716-4723, 2011.

Crossref

[5]

Bansal,

; Daniilidis,

Geometric urban geo-localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3978-3985, 2014.

Crossref

[6]

Kendall,

; Grimes,

; Cipolla,

PoseNet: A convolutional network for real-time 6-DOF camera relocalization. In: Proceedings of the IEEE International Conference on Computer Vision, 2938-2946, 2015.

Crossref

[7]

Walch,

; Hazirbas,

; Leal-Taixé,

; Sattler,

; Hilsenbeck,

; Cremers,

Image-based localization using LSTMs for structured feature correlation. In: Proceedings of the IEEE International Conference on Computer Vision, 627-637, 2017.

Crossref

[8]

Melekhov,

; Ylioinas,

; Kannala,

; Rahtu,

Image-based localization using hourglass networks. arXiv preprint arXiv:1703.07971, 2017.

Google Scholar

[9]

Sattler,

; Torii,

; Sivic,

; Pollefeys,

; Taira,

; Okutomi,

; Pajdla,

Are large-scale 3D models really necessary for accurate visual localization? In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6175-6184, 2017.

Crossref

[10]

Sivic,

; Zisserman,

Video Google: A text retrieval approach to object matching in videos. In: Proceedings 9th IEEE International Conference on Computer Vision, 1470-1477, 2003.

Crossref

[11]

Robertsone,

; Cipolla,

An Image-based system for urban navigation. In: Proceedings of the British Machine Conference, 84.1-84.10, 2004.

Crossref

[12]

Hays,

; Efros,

A. A.

IM2GPS: Estimating geographic information from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, 1-8, 2008.

Crossref

[13]

Bergamo,

; Sinha,

S. N.

; Torresani,

Leveraging structure from motion to learn discriminative codebooks for scalable landmark classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 763-770, 2013.

Crossref

[14]

Zhang,

; Kosecka,

Image based localization in urban environments. In: Proceedings of the 3rd International Symposium on 3D Data Processing, Visualization, and Transmission, 33-40, 2006.

Crossref

[15]

Nister,

; Stewenius,

Scalable recognition with a vocabulary tree. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2161-2168, 2006.

[16]

Schindler,

; Brown,

; Szeliski,

City-scale location recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1-7, 2007.

Crossref

[17]

Irschara,

; Zach,

; Frahm,

; Bischof,

From structure-from-motion point clouds to fast location recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2599-2606, 2009.

Crossref

[18]

Sattler,

; Leibe,

; Kobbelt,

Fast image-based localization using direct 2D-to-3D matching. In: Proceedings of the International Conference on Computer Vision, 667-674, 2011.

Crossref

[19]

Matei,

B. C.

; Vander Valk,

; Zhu,

; Cheng,

; Sawhney,

H. S.

Image to LIDAR matching for geotagging in urban environments. In: Proceedings of the IEEE Workshop on Applications of Computer Vision, 413-420, 2013.

Crossref

[20]

Svärm,

; Enqvist,

; Oskarsson,

; Kahl,

Accurate localization and pose estimation for large 3D models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 532-539, 2014.

Crossref

[21]

Baatz,

; Saurer,

; Köser,

; Pollefeys,

Large scale visual geo-localization of images in mountainous terrain. In: Computer Vision - ECCV 2012. Lecture Notes in Computer Science, Vol. 7573. Fitzgibbon,

; Lazebnik,

; Perona,

; Sato,

; Schmid,

Eds. Springer Berlin Heidelberg, 517-530, 2012.

Crossref

[22]

Svarm,

; Enqvist,

; Kahl,

; Oskarsson,

City-scale localization for cameras with known vertical direction. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 39, No. 7, 1455-1461, 2017.

Crossref Google Scholar

[23]

Piasco,

; Sidibé,

; Demonceaux,

; Gouet-Brunet,

A survey on visual-based localization: On the benefit of heterogeneous data. Pattern Recognition Vol. 74, 90-109, 2018.

Crossref Google Scholar

[24]

Szegedy,

; Liu,

; Jia,

; Sermanet,

; Reed,

; Anguelov,

; Erhan,

; Vanhoucke,

; Rabinovich,

Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1-9, 2015.

Crossref

[25]

He,

; Zhang,

; Ren,

; Sun,

Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770-778, 2016.

Crossref

[26]

Kendall,

; Cipolla,

Modelling uncertainty in deep learning for camera relocalization. In: Proceedings of the IEEE International Conference on Robotics and Automation, 4762-4769, 2016.

Crossref

[27]

Kendall,

; Cipolla,

Geometric loss functions for camera pose regression with deep learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6555-6564, 2017.

Crossref

[28]

Berlin Partner für Wirtschaft und Technologie GmbH. Berlin 3D city model. 2016. Available at https://www.businesslocationcenter.de/en/WA/B/seite0.jsp.

[29]

Zhang,

; Bengio,

; Hardt,

; Recht,

; Vinyals,

Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.

Google Scholar

[30]

Russakovsky,

; Deng,

; Su,

; Krause,

; Satheesh,

; Ma,

; Huang,

; Karpathy,

; Khosla,

; Bernstein,

; Berg,

A. C.

; Fei-Fei,

ImageNet large scale visual recognition challenge. International Journal of Computer Vision Vol. 115, No. 3, 211-252, 2015.

Crossref Google Scholar

[31]

Deng,

; Dong,

; Socher,

; Li,

L.-J.

; Li,

; Fei-Fei,

ImageNet: A large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 248-255, 2009.

Crossref

[32]

OpenStreetMap Wiki contributors. OSM-3D.org.OpenStreetMap Wiki, 2018. Available athttps://wiki.openstreetmap.org/w/index.php?title=OSM-3D.org&oldid=2025859.

Computational Visual Media

Volume 7 Issue 1,
March 2021

Pages 103-113

DOI: 10.1007/s41095-020-0196-2

Cite this article:

Kadosh M, Moses Y, Shamir A. On the role of geometry in geo-localization. Computational Visual Media, 2021, 7(1): 103-113. https://doi.org/10.1007/s41095-020-0196-2

Return

10.1007/s41095-020-0196-2.T001Table 1Experimental results. We give the proportion of images for which a correct estimation was obtained, relative to the total number of valid images evaluated (higher is better). For geo-matching we use the nearest neighbor measure (nn), and for geo-interpolation, the Manhattan distance $(D)$

	Input type	Geo-matching			Geo-interpolation
		(B) Arbitrary pose	(A) Correct pose		(C) Correct pose
		1nn	1nn	3nn	2D $(x, y)$		4D $(x, y, θ, ϕ)$
		1nn	1nn	3nn	$D$ <1	$D$ <3	$D$ <1	$D$ <3
Area 400 $\times$ 400step 2037k images	Edges	0.45	0.97	0.99	0.64	0.82	0.58	0.75
	Faces	0.35	0.99	0.99	0.56	0.76	0.51	0.69
	Depth	0.23	0.99	0.99	0.61	0.79	0.55	0.72
	Edges+Faces	0.29	0.98	0.99	0.72	0.88	0.65	0.82
	Edges+Faces+Depth	0.24	0.98	0.99	0.71	0.88	0.64	0.81
Area 400 $\times$ 400step 10140k images	Edges	0.11	0.98	0.98	0.85	0.94	0.84	0.93
	Faces	0.05	0.97	0.97	0.80	0.90	0.79	0.88
	Depth	0.06	0.97	0.97	0.83	0.92	0.82	0.91
	Edges+Faces	0.09	0.97	0.97	0.88	0.96	0.87	0.95
	Edges+Faces+Depth	0.08	0.94	0.95	0.88	0.96	0.87	0.95
Area 800 $\times$ 800step 20170k images	Edges	0.06	0.96	0.96	0.62	0.78	0.59	0.75
	Faces	0.01	0.96	0.96	0.51	0.68	0.48	0.65
	Depth	0.01	0.96	0.97	0.61	0.77	0.59	0.73
	Edges+Faces	0.04	0.92	0.93	0.70	0.86	0.67	0.83
	Edges+Faces+Depth	0.03	0.95	0.96	0.70	0.85	0.67	0.81

10.1007/s41095-020-0196-2.T002Table 2Examples of $ℓ_{2}$ errors for an experiment with Edges+Faces image type in each sub-space; spatial $(x, y)$ errors in meters, and orientation $(θ, ϕ)$ ) errors in degrees. As in this example, usually the errors show a long-tail distribution: many images have small errors and a few have very large errors

	(A) Geo-matching				(C) Geo-interpolation
	$(x, y)$		$(θ, ϕ)$		$(x, y)$		$(θ, ϕ)$
	mean	median	mean	median	mean	median	mean	median
Area 400×400step 2037k images	3.65	3.26	0.84	0.69	26.30	11.26	10.95	1.84
Area 400 $\times$ 400step 10140k images	2.37	2.10	0.57	0.48	7.99	3.67	2.65	0.67
Area 800 $\times$ 800step 20170k images	5.43	4.71	0.67	0.54	40.23	12.28	9.80	1.40

10.1007/s41095-020-0196-2.T003Table 3Testing network learning capacity: results from a single experiment with edge image input. The network’s ability to learn drops when the number of images grows beyond a certain point

	(A) Geo-matching		(C) Geo-interpolation
	1nn	3nn	2D $(x, y)$		4D $(x, y, θ, ϕ)$
	1nn	3nn	$D$ <1	$D$ <3	$D$ <1	$D$ <3
Area 800 $\times$ 800step 10636k images	0.82	0.82	0.80	0.92	0.79	0.92
Area 1600 $\times$ 1600step 20666k images	0.58	0.59	0.46	0.69	0.44	0.67

10.1007/s41095-020-0196-2.T004Table 4Low grid density results. Datasets (single experiment each) with sparse spatial sampling (top two blocks), and sparse spatial and orientation sampling (bottom two blocks) where pitch $= 6^{\circ}, 7^{\circ}, \dots, 12^{\circ}$ , and yaw $= 0^{\circ}, 45^{\circ}, \dots, 315^{\circ}$ . The sparser the data, the worse the results. Geo-interpolation failed for very sparse and very small datasets

	Input type	(A) Geo-matching		(C) Geo-interpolation
		1nn	3nn	2D $(x, y)$		4D $(x, y, θ, ϕ)$
		1nn	3nn	$D$ <1	$D$ <3	$D$ <1	$D$ <3
Area 800×800	Edges	0.90	0.96	0.39	0.62	0.30	0.49
step 40	Edges+Faces	0.91	0.97	0.48	0.72	0.38	0.59
61k images	Edges+Faces+Depth	0.96	0.98	0.48	0.72	0.38	0.58
Area 1600 $\times$ 1600	Edges	0.89	0.90	0.22	0.39	0.19	0.32
step 40	Edges+Faces	0.94	0.95	0.30	0.50	0.24	0.41
174k images	Edges+Faces+Depth	0.94	0.96	0.32	0.51	0.26	0.43
Area 800 $\times$ 800	Edges	0.40	0.41
step 40 / sparse angles	Edges+Faces	0.37	0.38	Failed
2.5k images	Edges+Faces+Depth	0.26	0.27
Area 1600 $\times$ 1600	Edges	0.16	0.18
step 40 / sparse angles	Edges+Faces	0.17	0.19	Failed
7k images	Edges+Faces+Depth	0.13	0.14