PDF (7 MB)
Collect
Submit Manuscript
Research Article | Open Access | Online First

Enhancing construction robot collaboration via multiagent reinforcement learning

Kangkang DuanaZhengbo Zoub()
Department of Civil Engineering, The University of British Columbia, Vancouver V6T 1Z4, Canada
Department of Civil Engineering and Engineering Mechanics, Columbia University, New York 10025, USA
Show Author Information

Abstract

The construction industry necessitates complex interactions among multiple agents (e.g., workers and robots) for efficient task execution. In this paper, we present a framework that aims at achieving multiagent reinforcement learning (RL) for robot control in construction tasks. Our proposed framework leverages the principles of proximal policy optimization (PPO) and develops a multiagent variant to enable robots to acquire sophisticated control policies. We evaluated the effectiveness of our framework through four collaborative construction tasks. The results revealed the efficient collaboration mechanism between agents and demonstrated the ability of our approach to enable multiple robots to learn and adapt their behaviors in complex and dynamic construction tasks while effectively preventing collisions. The results also revealed the advantages of combining RL and inverse kinematics (IK) in enabling precise installation. The findings from this research contribute to the advancement of multiagent RL in the domain of construction robotics. By enabling robots to collaborate effectively, we pave the way for more efficient, flexible, and intelligent construction processes.

References

[1]

H. Golizadeh, C. K. H. Hon, R. Drogemuller, et al. Digital engineering potential in addressing causes of construction accidents. Automat Constr, 2018, 95: 284–295.

[2]

X. Y. Huang, J. Hinze. Analysis of construction worker fall accidents. J Constr Eng Manage, 2003, 129: 262–271.

[3]

Q. P. Ha, L. Yen, C. Balaguer. Robotic autonomous systems for earthmoving in military applications. Automat Constr, 2019, 107: 102934.

[4]

E. Gambao, C. Balaguer, F. Gebhart. Robot assembly system for computer-integrated construction. Automat Constr, 2000, 9: 479–487.

[5]

K. Jung, B. Chu, D. Hong. Robot-based construction automation: An application to steel beam assembly (Part II). Automat Constr, 2013, 32: 62–79.

[6]

B. Chu, K. Jung, M. T. Lim, et al. Robot-based construction automation: An application to steel beam assembly (Part I). Automat Constr, 2013, 32: 46–61.

[7]

X. Y. Ma, C. Mao, G. W. Liu. Can robots replace human beings?—Assessment on the developmental potential of construction robot. J Build Eng, 2022, 56: 104727.

[8]

C. J. Liang, X. Wang, V. R. Kamat, et al. Human–robot collaboration in construction: Classification and research trends. J Constr Eng Manage, 2021, 147: 03121006.

[9]

M. Vega-Heredia, R. E. Mohan, T. Y. Wen, et al. Design and modelling of a modular window cleaning robot. Automat Constr, 2019, 103: 268–278.

[10]

R. Saltaren, R. Aracil, O. Reinoso. Analysis of a climbing parallel robot for construction applications. Comput Aided Civ Inf, 2004, 19: 436–445.

[11]

N. Melenbrink, J. Werfel, A. Menges. On-site autonomous construction robots: Towards unsupervised building. Automat Constr, 2020, 119: 103312.

[12]

N. Melenbrink, K. Rinderspacher, A. Menges, et al. Autonomous anchoring for robotic construction. Automat Constr, 2020, 120: 103391.

[13]

S. K. Baduge, S. Thilakarathna, J. S. Perera, et al. Artificial intelligence and smart vision for building and construction 4.0: Machine and deep learning methods and applications. Automat Constr, 2022, 141: 104440.

[14]

T. Sasaki, K. Kawashima. Remote control of backhoe at construction site with a pneumatic robot system. Automat Constr, 2008, 17: 907–914.

[15]

S. Lee, T. M. Adams. Spatial model for path planning of multiple mobile construction robots. Comput Aided Civ Inf, 2004, 19: 231–245.

[16]

K. S. Holkar, M. L. Waghmare. An overview of model predictive control. Int J Control Autom, 2010, 3: 47–64.

[17]
J. van den Berg, M. Lin, D. Manocha. Reciprocal velocity obstacles for real-time multi-agent navigation. In: Proceedings of 2008 IEEE International Conference on Robotics and Automation, Pasadena, USA, 2008.
[18]

M. Ambrosino, F. Boucher, P. Mengeot, et al. A trajectory-based explicit reference governor for the laying activity with heavy pre-fabricated elements. Constr Robot, 2023, 7: 41–52.

[19]

M. M. Nicotra, E. Garone. The explicit reference governor: A general framework for the closed-form control of constrained nonlinear systems. IEEE Control Syst Mag, 2018, 38: 89–107.

[20]

K. Merckaert, B. Convens, C. J. Wu, et al. Real-time motion control of robotic manipulators for safe human–robot coexistence. Rob Comput Integr Manuf, 2022, 73: 102223.

[21]

D. Lee, M. Kim. Autonomous construction hoist system based on deep reinforcement learning in high-rise building construction. Automat Constr, 2021, 128: 103737.

[22]
R. S. Sutton, A. G. Barto. Reinforcement Learning: An Introduction. 2nd ed. Cambridge, USA: The MIT Press, 2018.
[23]
A. Nagabandi, G. Kahn, R. S. Fearing, et al. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In: Proceedings of 2018 IEEE International Conference on Robotics and Automation, Brisbane, Australia, 2018.
[24]
V. Mnih, K. Kavukcuoglu, D. Silver, et al. Playing Atari with deep reinforcement learning. 2013, arXiv:1312.5602. arXiv.org e-print archive. https://arxiv.org/abs/1312.5602 (accessed 2025-01-08
[25]
J. Schulman, F. Wolski, P. Dhariwal, et al. Proximal policy optimization algorithms. 2017, arXiv:1707.06347. arXiv. org e-print archive. https://arxiv.org/abs/1707.06347 (accessed 2025-01-08
[26]
T. P. Lillicrap, J. J. Hunt, A. Pritzel, et al. Continuous control with deep reinforcement learning. In: Proceedings of the 4th International Conference on Learning Representations, San Juan, Puerto Rico, 2016.
[27]

B. Belousov, B. Wibranek, J. Schneider, et al. Robotic architectural assembly with tactile skills: Simulation and optimization. Automat Constr, 2022, 133: 104006.

[28]

A. A. Apolinarska, M. Pacher, H. Li, et al. Robotic assembly of timber joints using reinforcement learning. Automat Constr, 2021, 125: 103569.

[29]
S. R. B. dos Santos, S. N. Givigi, C. L. Nascimento. Autonomous construction of structures in a dynamic environment using Reinforcement Learning. In: Proceedings of 2013 IEEE International Systems Conference, Orlando, USA, 2013: pp 452–459.
[30]

H. P. Li, H. B. He. Multiagent trust region policy optimization. IEEE Trans Neural Netw Learn Syst, 2024, 35: 12873–12887.

[31]
R. Lowe, Y. Wu, A. Tamar, et al. Multi-agent actor–critic for mixed cooperative–competitive environments. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, 2020: pp 6382–6393.
[32]

Y. D. Wang, H. Liu, W. B. Zheng, et al. Multi-objective workflow scheduling with deep- Q-network-based multi-agent reinforcement learning. IEEE Access, 2019, 7: 39974–39982.

[33]

T. Rashid, M. Samvelyan, C. S. de Witt, et al. Monotonic value function factorisation for deep multi-agent reinforcement learning. J Mach Learn Res, 2020, 21: 178.

[34]

H. S. Choi, C. S. Han, K. Y. Lee, et al. Development of hybrid robot for construction works with pneumatic actuator. Automat Constr, 2005, 14: 452–459.

[35]

S. Kang, E. Miranda. Planning and visualization for automated robotic crane erection processes in construction. Automat Constr, 2006, 15: 398–414.

[36]
V. R. Konda, J. N. Tsitsiklis. Actor–critic algorithms. In: Proceedings of the 12th Advances in Neural Information Processing Systems, Denver, USA, 2000.
[37]
V. Mnih, A. P. Badia, M. Mirza, et al. Asynchronous methods for deep reinforcement learning. In: Proceedings of the 33rd International Conference on Machine Learning, New York City, USA, 2016.
[38]
F. Torabi, G. Warnell, P. Stone. Behavioral cloning from observation. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 2018.
[39]
J. Ho, S. Ermon. Generative adversarial imitation learning. In: Proceedings of the 30th Conference on Neural Information Processing Systems, Barcelona, Spain, 2016.
[40]

S. Cho, S. Han. Reinforcement learning-based simulation and automation for tower crane 3D lift planning. Automat Constr, 2022, 144: 104620.

[41]

F. A. Azad, S. A. Rad, M. Arashpour. Back-stepping control of delta parallel robots with smart dynamic model selection for construction applications. Automat Constr, 2022, 137: 104211.

[42]

J. N. Cai, A. Du, X. Y. Liang, et al. Prediction-based path planning for safe and efficient human–robot collaboration in construction via deep reinforcement learning. J Comput Civ Eng, 2023, 37: 04022046.

[43]

C. J. Liang, V. R. Kamat, C. C. Menassa. Teaching robots to perform quasi-repetitive construction tasks through human demonstration. Automat Constr, 2020, 120: 103370.

[44]

L. Huang, Z. H. Zhu, Z. B. Zou. To imitate or not to imitate: Boosting reinforcement learning-based construction robotic control for long-horizon tasks using virtual demonstrations. Automat Constr, 2023, 146: 104691.

[45]
K. K. Duan, Z. B. Zou, T. Y. Yang. Training of construction robots using imitation learning and environmental rewards. Comput Aided Civ Inf, 2024, in press. https://doi.org/10.1111/mice.13394 (accessed 2025-01-08
[46]

V. Asghari, A. J. Biglari, S. C. Hsu. Multiagent reinforcement learning for project-level intervention planning under multiple uncertainties. J Manage Eng, 2023, 39: 04022075.

[47]

C. P. Andriotis, K. G. Papakonstantinou. Managing engineering systems with large state and action spaces through deep reinforcement learning. Reliab Eng Syst Safe, 2019, 191: 106483.

[48]

L. Yu, Z. B. Xu, T. F. Zhang, et al. Energy-efficient personalized thermal comfort control in office buildings based on multi-agent deep reinforcement learning. Build Environ, 2022, 223: 109458.

[49]
J. Vazquez-Canteli, T. Detjeen, G. Henze, et al. Multi-agent reinforcement learning for adaptive demand response in smart cities. J Phys: Conf Ser, 2019, 1343: 012058.
[50]
K. Miyazaki, N. Matsunaga, K. Murata. Formation path learning for cooperative transportation of multiple robots using MADDPG. In: Proceedings of the 21st International Conference on Control, Automation and Systems, Jeju, Korea, 2021: pp 1619–1623.
[51]

J. P. Liu, P. K. Liu, L. Feng, et al. Automated clash resolution for reinforcement steel design in concrete frames via Q-learning and Building Information Modeling. Automat Constr, 2020, 112: 103062.

[52]

J. Zhong, T. Wang, L. L. Cheng. Collision-free path planning for welding manipulator via hybrid algorithm of deep reinforcement learning and inverse kinematics. Complex Intell Syst, 2022, 8: 1899–1912.

[53]

G. Elías Alonso, X. G. Jin. Skeleton‐level control for multi‐agent simulation through deep reinforcement learning. Comput Animat Virt Worlds, 2022, 33: e2079.

[54]
C. R. Garrett. PyBullet planning [Online]. https://pypi.org/project/pybullet-planning/ (accessed 2025-01-08).
[55]
R. Diankov. Automated construction of robotic manipulation programs. Ph.D. Thesis, Schenley Park Pittsburgh, USA: Carnegie Mellon University, 2010.
[56]
S. Y. Huang, S. Ontañón. A closer look at invalid action masking in policy gradient algorithms. 2022, arXiv: 2006.14171. arXiv.org e-print archive. https://arxiv.org/abs/2006.14171 (accessed 2025-01-08
[57]
C. S. de Witt, T. Gupta, D. Makoviichuk, et al. Is independent learning all you need in the StarCraft multi-agent challenge? 2020, arXiv:2011.09533, arXiv.org e-print archive. https://arxiv.org/abs/2011.09533 (accessed 2025-01-08
[58]
C. Yu, A. Velu, E. Vinitsky, et al. The surprising effectiveness of PPO in cooperative multi-agent games. In: Proceedings of the 36th International Conference on Neural Information Processing Systems, New Orleans, USA, 2022.
[59]
J. Schulman, P. Moritz, S. Levine, et al. High-dimensional continuous control using generalized advantage estimation. 2018, arXiv:1506.02438. arXiv.org e-print archive. https://arxiv.org/abs/1506.02438 (accessed 2025-01-08
[60]
E. Coumans. Bullet physics simulation. In: Proceedings of the ACM SIGGRAPH '15, Los Angeles, USA, 2015.
Journal of Intelligent Construction
Cite this article:
Duan K, Zou Z. Enhancing construction robot collaboration via multiagent reinforcement learning. Journal of Intelligent Construction, 2025, https://doi.org/10.26599/JIC.2025.9180089
Metrics & Citations  
Article History
Copyright
Rights and Permissions
Return