Home Journal of Intelligent Construction Article
PDF (7 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript
Research Article | Open Access | Online First

Enhancing construction robot collaboration via multiagent reinforcement learning

Kangkang Duana, Zhengbo Zoub()
Department of Civil Engineering, The University of British Columbia, Vancouver V6T 1Z4, Canada
Department of Civil Engineering and Engineering Mechanics, Columbia University, New York 10025, USA
Show Author Information

Abstract

The construction industry necessitates complex interactions among multiple agents (e.g., workers and robots) for efficient task execution. In this paper, we present a framework that aims at achieving multiagent reinforcement learning (RL) for robot control in construction tasks. Our proposed framework leverages the principles of proximal policy optimization (PPO) and develops a multiagent variant to enable robots to acquire sophisticated control policies. We evaluated the effectiveness of our framework through four collaborative construction tasks. The results revealed the efficient collaboration mechanism between agents and demonstrated the ability of our approach to enable multiple robots to learn and adapt their behaviors in complex and dynamic construction tasks while effectively preventing collisions. The results also revealed the advantages of combining RL and inverse kinematics (IK) in enabling precise installation. The findings from this research contribute to the advancement of multiagent RL in the domain of construction robotics. By enabling robots to collaborate effectively, we pave the way for more efficient, flexible, and intelligent construction processes.

Keywords

construction robotsmultiagent reinforcement learningtrajectory planning

References

[1]

H. Golizadeh, C. K. H. Hon, R. Drogemuller, et al. Digital engineering potential in addressing causes of construction accidents. Automat Constr, 2018, 95: 284–295.

Crossref Google Scholar
[2]

X. Y. Huang, J. Hinze. Analysis of construction worker fall accidents. J Constr Eng Manage, 2003, 129: 262–271.

Crossref Google Scholar
[3]

Q. P. Ha, L. Yen, C. Balaguer. Robotic autonomous systems for earthmoving in military applications. Automat Constr, 2019, 107: 102934.

Crossref Google Scholar
[4]

E. Gambao, C. Balaguer, F. Gebhart. Robot assembly system for computer-integrated construction. Automat Constr, 2000, 9: 479–487.

Crossref Google Scholar
[5]

K. Jung, B. Chu, D. Hong. Robot-based construction automation: An application to steel beam assembly (Part II). Automat Constr, 2013, 32: 62–79.

Crossref Google Scholar
[6]

B. Chu, K. Jung, M. T. Lim, et al. Robot-based construction automation: An application to steel beam assembly (Part I). Automat Constr, 2013, 32: 46–61.

Crossref Google Scholar
[7]

X. Y. Ma, C. Mao, G. W. Liu. Can robots replace human beings?—Assessment on the developmental potential of construction robot. J Build Eng, 2022, 56: 104727.

Crossref Google Scholar
[8]

C. J. Liang, X. Wang, V. R. Kamat, et al. Human–robot collaboration in construction: Classification and research trends. J Constr Eng Manage, 2021, 147: 03121006.

Crossref Google Scholar
[9]

M. Vega-Heredia, R. E. Mohan, T. Y. Wen, et al. Design and modelling of a modular window cleaning robot. Automat Constr, 2019, 103: 268–278.

Crossref Google Scholar
[10]

R. Saltaren, R. Aracil, O. Reinoso. Analysis of a climbing parallel robot for construction applications. Comput Aided Civ Inf, 2004, 19: 436–445.

Crossref Google Scholar
[11]

N. Melenbrink, J. Werfel, A. Menges. On-site autonomous construction robots: Towards unsupervised building. Automat Constr, 2020, 119: 103312.

Crossref Google Scholar
[12]

N. Melenbrink, K. Rinderspacher, A. Menges, et al. Autonomous anchoring for robotic construction. Automat Constr, 2020, 120: 103391.

Crossref Google Scholar
[13]

S. K. Baduge, S. Thilakarathna, J. S. Perera, et al. Artificial intelligence and smart vision for building and construction 4.0: Machine and deep learning methods and applications. Automat Constr, 2022, 141: 104440.

Crossref Google Scholar
[14]

T. Sasaki, K. Kawashima. Remote control of backhoe at construction site with a pneumatic robot system. Automat Constr, 2008, 17: 907–914.

Crossref Google Scholar
[15]

S. Lee, T. M. Adams. Spatial model for path planning of multiple mobile construction robots. Comput Aided Civ Inf, 2004, 19: 231–245.

Crossref Google Scholar
[16]

K. S. Holkar, M. L. Waghmare. An overview of model predictive control. Int J Control Autom, 2010, 3: 47–64.

Google Scholar
[17]
J. van den Berg, M. Lin, D. Manocha. Reciprocal velocity obstacles for real-time multi-agent navigation. In: Proceedings of 2008 IEEE International Conference on Robotics and Automation, Pasadena, USA, 2008.
Crossref
[18]

M. Ambrosino, F. Boucher, P. Mengeot, et al. A trajectory-based explicit reference governor for the laying activity with heavy pre-fabricated elements. Constr Robot, 2023, 7: 41–52.

Crossref Google Scholar
[19]

M. M. Nicotra, E. Garone. The explicit reference governor: A general framework for the closed-form control of constrained nonlinear systems. IEEE Control Syst Mag, 2018, 38: 89–107.

Crossref Google Scholar
[20]

K. Merckaert, B. Convens, C. J. Wu, et al. Real-time motion control of robotic manipulators for safe human–robot coexistence. Rob Comput Integr Manuf, 2022, 73: 102223.

Crossref Google Scholar
[21]

D. Lee, M. Kim. Autonomous construction hoist system based on deep reinforcement learning in high-rise building construction. Automat Constr, 2021, 128: 103737.

Crossref Google Scholar
[22]
R. S. Sutton, A. G. Barto. Reinforcement Learning: An Introduction. 2nd ed. Cambridge, USA: The MIT Press, 2018.
[23]
A. Nagabandi, G. Kahn, R. S. Fearing, et al. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In: Proceedings of 2018 IEEE International Conference on Robotics and Automation, Brisbane, Australia, 2018.
Crossref
[24]
V. Mnih, K. Kavukcuoglu, D. Silver, et al. Playing Atari with deep reinforcement learning. 2013, arXiv:1312.5602. arXiv.org e-print archive. https://arxiv.org/abs/1312.5602 (accessed 2025-01-08
[25]
J. Schulman, F. Wolski, P. Dhariwal, et al. Proximal policy optimization algorithms. 2017, arXiv:1707.06347. arXiv. org e-print archive. https://arxiv.org/abs/1707.06347 (accessed 2025-01-08
[26]
T. P. Lillicrap, J. J. Hunt, A. Pritzel, et al. Continuous control with deep reinforcement learning. In: Proceedings of the 4th International Conference on Learning Representations, San Juan, Puerto Rico, 2016.
[27]

B. Belousov, B. Wibranek, J. Schneider, et al. Robotic architectural assembly with tactile skills: Simulation and optimization. Automat Constr, 2022, 133: 104006.

Crossref Google Scholar
[28]

A. A. Apolinarska, M. Pacher, H. Li, et al. Robotic assembly of timber joints using reinforcement learning. Automat Constr, 2021, 125: 103569.

Crossref Google Scholar
[29]
S. R. B. dos Santos, S. N. Givigi, C. L. Nascimento. Autonomous construction of structures in a dynamic environment using Reinforcement Learning. In: Proceedings of 2013 IEEE International Systems Conference, Orlando, USA, 2013: pp 452–459.
Crossref
[30]

H. P. Li, H. B. He. Multiagent trust region policy optimization. IEEE Trans Neural Netw Learn Syst, 2024, 35: 12873–12887.

Crossref Google Scholar
[31]
R. Lowe, Y. Wu, A. Tamar, et al. Multi-agent actor–critic for mixed cooperative–competitive environments. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, 2020: pp 6382–6393.
[32]

Y. D. Wang, H. Liu, W. B. Zheng, et al. Multi-objective workflow scheduling with deep- Q-network-based multi-agent reinforcement learning. IEEE Access, 2019, 7: 39974–39982.

Crossref Google Scholar
[33]

T. Rashid, M. Samvelyan, C. S. de Witt, et al. Monotonic value function factorisation for deep multi-agent reinforcement learning. J Mach Learn Res, 2020, 21: 178.

Crossref Google Scholar
[34]

H. S. Choi, C. S. Han, K. Y. Lee, et al. Development of hybrid robot for construction works with pneumatic actuator. Automat Constr, 2005, 14: 452–459.

Crossref Google Scholar
[35]

S. Kang, E. Miranda. Planning and visualization for automated robotic crane erection processes in construction. Automat Constr, 2006, 15: 398–414.

Crossref Google Scholar
[36]
V. R. Konda, J. N. Tsitsiklis. Actor–critic algorithms. In: Proceedings of the 12th Advances in Neural Information Processing Systems, Denver, USA, 2000.
[37]
V. Mnih, A. P. Badia, M. Mirza, et al. Asynchronous methods for deep reinforcement learning. In: Proceedings of the 33rd International Conference on Machine Learning, New York City, USA, 2016.
[38]
F. Torabi, G. Warnell, P. Stone. Behavioral cloning from observation. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 2018.
Crossref
[39]
J. Ho, S. Ermon. Generative adversarial imitation learning. In: Proceedings of the 30th Conference on Neural Information Processing Systems, Barcelona, Spain, 2016.
[40]

S. Cho, S. Han. Reinforcement learning-based simulation and automation for tower crane 3D lift planning. Automat Constr, 2022, 144: 104620.

Crossref Google Scholar
[41]

F. A. Azad, S. A. Rad, M. Arashpour. Back-stepping control of delta parallel robots with smart dynamic model selection for construction applications. Automat Constr, 2022, 137: 104211.

Crossref Google Scholar
[42]

J. N. Cai, A. Du, X. Y. Liang, et al. Prediction-based path planning for safe and efficient human–robot collaboration in construction via deep reinforcement learning. J Comput Civ Eng, 2023, 37: 04022046.

Crossref Google Scholar
[43]

C. J. Liang, V. R. Kamat, C. C. Menassa. Teaching robots to perform quasi-repetitive construction tasks through human demonstration. Automat Constr, 2020, 120: 103370.

Crossref Google Scholar
[44]

L. Huang, Z. H. Zhu, Z. B. Zou. To imitate or not to imitate: Boosting reinforcement learning-based construction robotic control for long-horizon tasks using virtual demonstrations. Automat Constr, 2023, 146: 104691.

Crossref Google Scholar
[45]
K. K. Duan, Z. B. Zou, T. Y. Yang. Training of construction robots using imitation learning and environmental rewards. Comput Aided Civ Inf, 2024, in press. https://doi.org/10.1111/mice.13394 (accessed 2025-01-08
Crossref
[46]

V. Asghari, A. J. Biglari, S. C. Hsu. Multiagent reinforcement learning for project-level intervention planning under multiple uncertainties. J Manage Eng, 2023, 39: 04022075.

Crossref Google Scholar
[47]

C. P. Andriotis, K. G. Papakonstantinou. Managing engineering systems with large state and action spaces through deep reinforcement learning. Reliab Eng Syst Safe, 2019, 191: 106483.

Crossref Google Scholar
[48]

L. Yu, Z. B. Xu, T. F. Zhang, et al. Energy-efficient personalized thermal comfort control in office buildings based on multi-agent deep reinforcement learning. Build Environ, 2022, 223: 109458.

Crossref Google Scholar
[49]
J. Vazquez-Canteli, T. Detjeen, G. Henze, et al. Multi-agent reinforcement learning for adaptive demand response in smart cities. J Phys: Conf Ser, 2019, 1343: 012058.
Crossref
[50]
K. Miyazaki, N. Matsunaga, K. Murata. Formation path learning for cooperative transportation of multiple robots using MADDPG. In: Proceedings of the 21st International Conference on Control, Automation and Systems, Jeju, Korea, 2021: pp 1619–1623.
Crossref
[51]

J. P. Liu, P. K. Liu, L. Feng, et al. Automated clash resolution for reinforcement steel design in concrete frames via Q-learning and Building Information Modeling. Automat Constr, 2020, 112: 103062.

Crossref Google Scholar
[52]

J. Zhong, T. Wang, L. L. Cheng. Collision-free path planning for welding manipulator via hybrid algorithm of deep reinforcement learning and inverse kinematics. Complex Intell Syst, 2022, 8: 1899–1912.

Crossref Google Scholar
[53]

G. Elías Alonso, X. G. Jin. Skeleton‐level control for multi‐agent simulation through deep reinforcement learning. Comput Animat Virt Worlds, 2022, 33: e2079.

Crossref Google Scholar
[54]
C. R. Garrett. PyBullet planning [Online]. https://pypi.org/project/pybullet-planning/ (accessed 2025-01-08).
[55]
R. Diankov. Automated construction of robotic manipulation programs. Ph.D. Thesis, Schenley Park Pittsburgh, USA: Carnegie Mellon University, 2010.
[56]
S. Y. Huang, S. Ontañón. A closer look at invalid action masking in policy gradient algorithms. 2022, arXiv: 2006.14171. arXiv.org e-print archive. https://arxiv.org/abs/2006.14171 (accessed 2025-01-08
[57]
C. S. de Witt, T. Gupta, D. Makoviichuk, et al. Is independent learning all you need in the StarCraft multi-agent challenge? 2020, arXiv:2011.09533, arXiv.org e-print archive. https://arxiv.org/abs/2011.09533 (accessed 2025-01-08
[58]
C. Yu, A. Velu, E. Vinitsky, et al. The surprising effectiveness of PPO in cooperative multi-agent games. In: Proceedings of the 36th International Conference on Neural Information Processing Systems, New Orleans, USA, 2022.
[59]
J. Schulman, P. Moritz, S. Levine, et al. High-dimensional continuous control using generalized advantage estimation. 2018, arXiv:1506.02438. arXiv.org e-print archive. https://arxiv.org/abs/1506.02438 (accessed 2025-01-08
[60]
E. Coumans. Bullet physics simulation. In: Proceedings of the ACM SIGGRAPH '15, Los Angeles, USA, 2015.
Crossref
Journal of Intelligent Construction
DOI: 10.26599/JIC.2025.9180089
Cite this article:
Duan K, Zou Z. Enhancing construction robot collaboration via multiagent reinforcement learning. Journal of Intelligent Construction, 2025, https://doi.org/10.26599/JIC.2025.9180089
Metrics & Citations  
Article History
Copyright
Rights and Permissions
Return