Deep panoramic depth prediction and completion for indoor scenes

Fig. 1 Different kinds of sparse depth. First image (from the left): depth map captured by structured-light sensors (Matterport Pro 3D camera) has lots of missing areas when rooms are large, surfaces are shiny or thin, and strong lighting is abundant. Second image: a depth map captured by a LiDAR setup (two Velodyne VPN-16 shifted of the vertical direction with different direction) has lots of valid information but only for a few stripes. Third image: depth information may also come from triangulated features in purely image-based pipelines; indoor environments, however, have lots of flat textureless surfaces, and reliable features, here detected from SIFT, may be very sparse. Fourth image: a typical input from low-cost structured light sensors with sparse measurements only for a small subset of the captured camera pixels; for synthetic training, a typical approach is to use a Bernouilli distribution to sparsify inputs [ 9 ].

Fig. 2 Network architecture. Our network is constituted by a single-branch encoder–decoder, which processes together the dense visual and sparse geometric data. A residual-gated encoder takes as input 4 channels (RGB + sparse depth) returning fused features at different resolution. Multi-resolution features are compressed, flattened, and passed to an MHSA-single layer module (i.e., bottleneck). Decoding proceeds symmetrically to the encoder, but without using gating, to reach the final output resolution.

Fig. 3 Qualitative results on Matterport3D-SD dataset [ 26 ]. Masked samples in the results are missing samples in the ground truth.

Fig. 4 Qualitative performance on S3D-SD with a LiDAR configuration with 32 beams and on real mobile LiDAR indoor capture. Qualitative results with the same setup of Table 2 . Our results are compared to the Huang et al. [ 36 ] approach trained with the same equirectangular augmented S3D-SD dataset with varying sparsity patterns.

Fig. 5 Qualitative performance on S3D-SD with different input depth sparsity patterns. Qualitative results using simulated input from low-cost depth cameras using Bernouilli sampling and simulated input from SfM/stero pipelines, using an SIFT detector to place samples. Our results are compared to the Huang et al. [ 36 ] approach trained with the same equirectangular S3D-SD dataset.

Fig. 6 Qualitative performance on S3D-SD by point cloud (PC). In these examples, 3D point clouds are obtained by unprojecting depth maps, using the same setting of Table 2 , and visualizing them from a standard point of view. Note how the proposed approach improves reconstruction especially in regions where clear geometric structures from the architectural layout are present.

Fig. 7 Mobile RGB+LiDAR setup. To test our approach on a real-world panoramic RGB+LiDAR acquisition, we exploit a backpacked mobile scanner equipped with a full-view panoramic camera for the RGB capture and two LiDAR heads for sparse depth capture. Ground-truth dense depth for each pose is provided by reprojecting data coming from multiple poses of a static scanner.

Fig. 8 Performance with variable sparsity level. The graph depicts the value of ${\delta }_1$ as a function of input depth sparsity for our method and for the best competing method [ 36 ]. Continuous lines represent models trained with our augmentation strategy. Dotted lines show the same models but trained without augmentation (i.e., $40$ degrees sparse coverage with 32 active beams).

Fig. 9 Bad case. Results on almost-outdoor environment. Sparse samples from outdoor part, not properly masked, negatively affect the whole reconstruction.