A computer vision and point cloud-based monitoring approach for automated construction tasks using full-scale robotized mobile cranes
), Abstract
Recent years have witnessed rapid development and contemporary trends in smart construction research owing to advances in machine learning algorithms, modern sensory systems, and robotic technologies. In this paper, a novel economical computer vision (CV) and point cloud-based monitoring framework is proposed to assist in the lifting and relocation of construction sources via mobile cranes on site. The proposed framework incorporates a multicamera approach to achieve multiple goals, such as three-dimensional (3D) vision-based real-time reconstruction, 3D localization of construction resources, and safety monitoring. To demonstrate the effectiveness of the proposed framework, field experiments were conducted on a full-scale mobile crane. The results show that the proposed monitoring system achieves real-time performance, which can successfully recognize construction resources and guide the crane to initialize the lifting position and avoid potential moving workers during motion execution.
Keywords
References
K. H. Petersen, N. Napp, R. Stuart-Smith, et al. A review of collective robotic construction. Sci Robot, 2019, 4: eaau8479.
O. W. Chong, J. S. Zhang, R. M. Voyles, et al. BIM-based simulation of construction robotics in the assembly process of wood frames. Autom Constr, 2022, 137: 104194.
W. T. Qiao, Z. X. Wang, D. Wang, et al. A new mortise and tenon timber structure and its automatic construction system. J Build Eng, 2021, 44: 103369.
P. Martinez, M. Al-Hussein, R. Ahmad. A scientometric analysis and critical review of computer vision applications for construction. Autom Constr, 2019, 107: 102947.
M. Memarzadeh, M. Golparvar-Fard, J. C. Niebles. Automated 2D detection of construction equipment and workers from site video streams using histograms of oriented gradients and colors. Autom Constr, 2013, 32: 24–37,
M. Golparvar-Fard, A. Heydarian, J. C. Niebles. Vision-based action recognition of earthmoving equipment using spatio–temporal features and support vector machine classifiers. Adv Eng Inform, 2013, 27: 652–663,
H. Son, H. Seong, H. Choi, et al. Real-time vision-based warning system for prevention of collisions between workers and heavy equipment. J Comput Civ Eng, 2019, 33: 4019029.
Q. Wang. Automatic checks from 3D point cloud data for safety regulation compliance for scaffold work platforms. Autom Constr, 2019, 104: 38–51.
X. C. Luo, H. Li, H. Wang, et al. Vision-based detection and visualization of dynamic workspaces. Autom Constr, 2019, 104: 1–13.
S. Siebert, J. Teizer. Mobile 3D mapping for surveying earthwork projects using an Unmanned Aerial Vehicle (UAV) system. Autom Constr, 2014, 41: 1–14.
J. Seo, S. Han, S. Lee, et al. Computer vision techniques for construction safety and health monitoring. Adv Eng Inf, 2015, 29: 239–251.
W. L. Fang, L. Y. Ding, P. E. D. Love, et al. Computer vision applications in construction safety assurance. Autom Constr, 2020, 110: 103013.
V. K. Reja, K. Varghese, Q. P. Ha. Computer vision-based construction progress monitoring. Autom Constr, 2022, 138: 104245.
M. S. Ur Rehman, M. T. Shafiq, F. Ullah. Automated computer vision-based construction progress monitoring: A systematic review. Buildings, 2022, 12: 1037.
M. Yang, C. K. Wu, Y. J. Guo, et al. Transformer-based deep learning model and video dataset for unsafe action identification in construction projects. Autom Constr, 2023, 146: 104703.
S. Halder, K. Afsari, E. Chiou, et al. Construction inspection & monitoring with quadruped robots in future human-robot teaming: A preliminary study. J Build Eng, 2023, 65: 105814.
A. S. Kulinan, M. Park, P. P. W. Aung, et al. Advancing construction site workforce safety monitoring through BIM and computer vision integration. Autom Constr, 2024, 158: 105227.
X. Pan, T. Y. Yang. Postdisaster image-based damage detection and repair cost estimation of reinforced concrete buildings using dual convolutional neural networks. Comput Aided Civil Infrastruct Eng, 2020, 35: 495–510.
C. B. Li, R. K. L. Su, X. Pan. Assessment of out-of-plane structural defects using parallel laser line scanning system. Comput Aided Civil Infrastruct Eng, 2024, 39: 834–851.
S. Tavasoli, X. Pan, T. Y. Yang. Real-time autonomous indoor navigation and vision-based damage assessment of reinforced concrete structures using low-cost nano aerial vehicles. J Build Eng, 2023, 68: 106193.
X. Pan, T. Y. Yang. 3D vision-based out-of-plane displacement quantification for steel plate structures using structure-from-motion, deep learning, and point-cloud processing. Comput. Aided Civil Infrastruct Eng, 2023, 38: 547–561.
X. Pan, T. Y. Yang, Y. F. Xiao, et al. Vision-based real-time structural vibration measurement through deep-learning-based detection and tracking methods. Eng Struct, 2023, 281: 115676.
X. Pan, S. Tavasoli, T. Y. Yang. Autonomous 3D vision-based bolt loosening assessment using micro aerial vehicles. Comput Aided Civil Infrastruct Eng, 2023, 38: 2443–2454.
X. Pan, T. Y. Yang. 3D vision-based bolt loosening assessment using photogrammetry, deep neural networks, and 3D point-cloud processing. J Build Eng, 2023, 70: 106326.
X. Pan, T. Y. Yang. Image-based monitoring of bolt loosening through deep-learning-based integrated detection and tracking. Comput Aided Civil Infrastruct Eng, 2022, 37: 1207–1222.
J. L. Heng, Y. Dong, L. Lai, et al. Digital twins-boosted intelligent maintenance of ageing bridge hangers exposed to coupled corrosion–fatigue deterioration. Autom Constr, 2024, 167: 105697.
D. Dais, İ. E. Bal, E. Smyrou, et al. Automatic crack classification and segmentation on masonry surfaces using convolutional neural networks and transfer learning. Autom Constr, 2021, 125: 103606.
H. Seo, A. D. Raut, C. Chen, et al. Multi-label classification and automatic damage detection of masonry heritage building through CNN analysis of infrared thermal imaging. Remote Sens, 2023, 15: 2517.
N. N. Wang, X. F. Zhao, P. Zhao, et al. Automatic damage detection of historic masonry buildings based on mobile deep learning. Autom Constr, 2019, 103: 53–66.
L. M. Dang, H. X. Wang, Y. F. Li, et al. Deep learning-based masonry crack segmentation and real-life crack length measurement. Constr Build Mater, 2022, 359: 129438.
J. Lee, J. M. Yu. Automatic surface damage classification developed based on deep learning for wooden architectural heritage. ISPRS Ann Photogramm Remote Sens Spatial Inf Sci, 2023, X-M-1-2023: 151–157.
D. G. Lowe. Distinctive image features from scale-invariant keypoints. Int J Comput Vision, 2004, 60: 91–110.
R. I. Hartley, P. Sturm. Triangulation. Comput Vision Image Understanding, 1997, 68: 146–157.
Y. F. Xiao, X. Pan, T. T. Y. Yang. Nonlinear backstepping hierarchical control of shake table using high-gain observer. Earthq Eng Struct Dyn, 2022, 51: 3347–3366.
H. C. Yao, P. Tan, T. Y. Yang, et al. Shake table real-time hybrid testing for shear buildings based on sliding mode acceleration control method. Structures, 2023, 52: 230–240.
H. C. Yao, P. Tan, T. Y. Yang, et al. Acceleration-based sliding mode hierarchical control algorithm for shake table tests. Earthq Eng Struct Dyn, 2021, 50: 3670–3691.
京公网安备11010802044758号