AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Tsinghua Science and Technology Article
PDF (6.7 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Open Access

Deep-EERA: DRL-Based Energy-Efficient Resource Allocation in UAV-Empowered Beyond 5G Networks

School of Electronic Engineering, Beijing University of Posts and Telecommunication, Beijing 1000876, China
Department of Information and Communication Engineering, Yeungnam University, Gyeongsan 38541, Republic of Korea
School of Information Engineering, Xi’an Eurasia University, Xi’an 710065, China
Department of Computer Science, Hanyang University, Seoul 04763, Republic of Korea
Show Author Information

Abstract

The rise of innovative applications, like online gaming, smart healthcare, and Internet of Things (IoT) services, has increased demand for high data rates and seamless connectivity, posing challenges for Beyond 5G (B5G) networks. There is a need for cost-effective solutions to enhance spectral efficiency in densely populated areas, ensuring higher data rates and uninterrupted connectivity while minimizing costs. Unmanned Aerial Vehicles (UAVs) as Aerial Base Stations (ABSs) offer a promising and cost-effective solution to boost network capacity, especially during emergencies and high-data-rate demands. Nevertheless, integrating UAVs into the B5G networks presents new challenges, including resource scarcity, energy efficiency, resource allocation, optimal power transmission control, and maximizing overall throughput. This paper presents a UAV-assisted B5G communication system where UAVs act as ABSs, and introduces the Deep Reinforcement Learning (DRL) based Energy Efficient Resource Allocation (Deep-EERA) mechanism. An efficient DRL-based Deep Deterministic Policy Gradient (DDPG) mechanism is introduced for optimal resource allocation with the twin goals of energy efficiency and average throughput maximization. The proposed Deep-EERA method learns optimal policies to conserve energy and enhance throughput within the dynamic and complex UAV-empowered B5G environment. Through extensive simulations, we validate the performance of the proposed approach, demonstrating that it outperforms other baseline methods in energy efficiency and throughput maximization.

Keywords

Deep Reinforcement Learning (DRL)Unmanned Aerial Vehicles (UAVs)resource allocationenergy efficiency5G and beyond network

References

[1]

A. Dogra, R. K. Jha, and S. Jain, A survey on beyond 5G network with the advent of 6G: Architecture and emerging technologies, IEEE Access, vol. 9, pp. 67512–67547, 2021.
Crossref Google Scholar
[2]
B. Li, Z. Fei, and Y. Zhang, UAV communications for 5G and beyond: Recent advances and future trends, IEEE Internet Things J., vol. 6, no. 2, pp. 2241–2263, 2019.
Crossref
[3]

K. Abbas, T. A. Khan, M. Afaq, and W. C. Song, Network slice lifecycle management for 5G mobile networks: An intent-based networking approach, IEEE Access, vol. 9, pp. 80128–80146, 2021.
Crossref Google Scholar
[4]

M. Mozaffari, W. Saad, M. Bennis, Y. H. Nam, and M. Debbah, A tutorial on UAVs for wireless networks: Applications, challenges, and open problems, IEEE Commun. Surv. Tutor., vol. 21, no. 3, pp. 2334–2360, 2019.
Crossref Google Scholar
[5]

L. Gupta, R. Jain, and G. Vaszkun, Survey of important issues in UAV communication networks, IEEE Commun. Surv. Tutor., vol. 18, no. 2, pp. 1123–1152, 2016.
Crossref Google Scholar
[6]
P. W. Khan, G. Xu, M. A. Latif, K. Abbas, and A. Yasin, UAV’s agricultural image segmentation predicated by Clifford geometric algebra, IEEE Access, vol. 7, pp. 38442–38450, 2019.
Crossref
[7]
S. Chandrasekharan, K. Gomez, A. Al-Hourani, S. Kandeepan, T. Rasheed, L. Goratti, L. Reynaud, D. Grace, I. Bucaille, T. Wirth, et al., Designing and implementing future aerial communication networks, IEEE communications magazine, vol. 54, no. 5, pp. 26−34, 2016.
Crossref
[8]

S. Chandrasekharan, K. Gomez, A. Al-Hourani, S. Kandeepan, T. Rasheed, L. Goratti, L. Reynaud, D. Grace, I. Bucaille, T. Wirth, et al., Designing and implementing future aerial communication networks, IEEE Commun. Mag., vol. 54, no. 5, pp. 26–34, 2016.
Crossref Google Scholar
[9]

Z. Xiao, P. Xia, and X. G. Xia, Enabling UAV cellular with millimeter-wave communication: Potentials and approaches, IEEE Commun. Mag., vol. 54, no. 5, pp. 66–73, 2016.
Crossref Google Scholar
[10]

A. Sharma, P. Vanjani, N. Paliwal, C. M. W. Basnayaka, D. N. K. Jayakody, H. C. Wang, and P. Muthuchidambaranathan, Communication and networking technologies for UAVs: A survey, J. Netw. Comput. Appl., vol. 168, p. 102739, 2020.
Crossref Google Scholar
[11]

P. W. Khan, K. Abbas, H. Shaiba, A. Muthanna, A. Abuarqoub, and M. Khayyat, Energy efficient computation offloading mechanism in multi-server mobile edge computing—an integer linear optimization approach, Electronics, vol. 9, no. 6, p. 1010, 2020.
Crossref Google Scholar
[12]
R. Shahzadi, M. Ali, H. Z. Khan, and M. Naeem, UAV assisted 5G and beyond wireless networks: A survey, J. Netw. Comput. Appl., vol. 189, p. 103114, 2021.
Crossref
[13]
Y. Gao, J. Cao, P. Wang, J. Wang, M. Zhao, S. Cheng, S. Hu, and W. Lu, UAV based 5G wireless networks: A practical solution for emergency communications, in Proc. 2020 XXXIIIrd General Assembly and Scientific Symp. Int. Union of Radio Science, Rome, Italy, 2020, pp. 1–4.
Crossref
[14]

Q. Wu, Y. Zeng, and R. Zhang, Joint trajectory and communication design for multi-UAV enabled wireless networks, IEEE Trans. Wirel. Commun., vol. 17, no. 3, pp. 2109–2121, 2018.
Crossref Google Scholar
[15]

A. M. Seid, G. O. Boateng, B. Mareri, G. Sun, and W. Jiang, Multi-agent DRL for task offloading and resource allocation in multi-UAV enabled IoT edge network, IEEE Trans. Netw. Serv. Manage., vol. 18, no. 4, pp. 4531–4547, 2021.
Crossref Google Scholar
[16]

Q. Wu, J. Xu, Y. Zeng, D. W. K. Ng, N. Al-Dhahir, R. Schober, and A. L. Swindlehurst, A comprehensive overview on 5G-andbeyond networks with UAVs: From communications to sensing and intelligence, IEEE J. Sel. Areas Commun., vol. 39, no. 10, pp. 2912–2945, 2021.
Crossref Google Scholar
[17]

K. Abbas, J. Hong, N. Van Tu, J. H. Yoo, and J. W. K. Hong, Autonomous DRL-based energy efficient VM consolidation for cloud data centers, Phys. Commun., vol. 55, p. 101925, 2022.
Crossref Google Scholar
[18]

H. Ye, G. Y. Li, and B. H. F. Juang, Deep reinforcement learning based resource allocation for V2V communications, IEEE Trans. Veh. Technol., vol. 68, no. 4, pp. 3163–3173, 2019.
Crossref Google Scholar
[19]

X. He, K. Wang, H. Huang, T. Miyazaki, Y. Wang, and S. Guo, Green resource allocation based on deep reinforcement learning in content-centric IoT, IEEE Trans. Emerg. Top. Comput., vol. 8, no. 3, pp. 781–796, 2020.
Crossref Google Scholar
[20]

X. Liao, J. Shi, Z. Li, L. Zhang, and B. Xia, A model-driven deep reinforcement learning heuristic algorithm for resource allocation in ultra-dense cellular networks, IEEE Trans. Veh. Technol., vol. 69, no. 1, pp. 983–997, 2020.
Crossref Google Scholar
[21]
H. Li, H. Gao, T. Lv, and Y. Lu, Deep Q-learning based dynamic resource allocation for self-powered ultra-dense networks, in Proc. 2018 IEEE Int. Conf. Communications Workshops (ICC Workshops ), Kansas City, MO, USA, 2018, pp. 1–6.
Crossref
[22]

Z. Wang, V. Aggarwal, and X. Wang, Power allocation for energy harvesting transmitter with causal information, IEEE Trans. Commun., vol. 62, no. 11, pp. 4080–4093, 2014.
Crossref Google Scholar
[23]

H. Zhang, S. Huang, C. Jiang, K. Long, V. C. M. Leung, and H. V. Poor, Energy efficient user association and power allocation in millimeter-wave-based ultra dense networks with energy harvesting base stations, IEEE J. Sel. Areas Commun., vol. 35, no. 9, pp. 1936–1947, 2017.
Crossref Google Scholar
[24]
R. S. Sutton and A. G. Barto, Reinforcement Learning : An Introduction, 2nd ed. Cambridge, MA, USA: MIT Press, 2018.
[25]

C. Savaglio, P. Pace, G. Aloi, A. Liotta, and G. Fortino, Lightweight reinforcement learning for energy efficient communications in wireless sensor networks, IEEE Access, vol. 7, pp. 29355–29364, 2019.
Crossref Google Scholar
[26]

Z. M. Fadlullah, F. Tang, B. Mao, N. Kato, O. Akashi, T. Inoue, and K. Mizutani, State-of-the-art deep learning: Evolving machine intelligence toward tomorrow’s intelligent network traffic control systems, IEEE Commun. Surv. Tutor., vol. 19, no. 4, pp. 2432–2455, 2017.
Crossref Google Scholar
[27]
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, Playing atari with deep reinforcement learning, arXiv preprint arXiv: 1312.5602, 2013.
[28]

X. Li, J. Fang, W. Cheng, H. Duan, Z. Chen, and H. Li, Intelligent power control for spectrum sharing in cognitive radios: A deep reinforcement learning approach, IEEE Access, vol. 6, pp. 25463–25473, 2018.
Crossref Google Scholar
[29]

Y. Du, F. Zhang, and L. Xue, A kind of joint routing and resource allocation scheme based on prioritized memories-deep Q network for cognitive radio ad hoc networks, Sensors, vol. 18, no. 7, p. 2119, 2018.
Crossref Google Scholar
[30]

M. Chu, H. Li, X. Liao, and S. Cui, Reinforcement learning-based multiaccess control and battery prediction with energy harvesting in IoT systems, IEEE Internet Things J., vol. 6, no. 2, pp. 2009–2020, 2019.
Crossref Google Scholar
[31]

M. Mohammadi, A. Al-Fuqaha, M. Guizani, and J. S. Oh, Semisupervised deep reinforcement learning in support of IoT and smart city services, IEEE Internet Things J., vol. 5, no. 2, pp. 624–635, 2018.
Crossref Google Scholar
[32]
B. Omoniwa, B. Galkin and I. Dusparic, Optimizing energy efficiency in UAV-assisted networks using deep reinforcement learning, IEEE Wireless Communications Letters, vol. 11, no. 8, pp. 1590−1594, 2022.
Crossref
[33]

Y. Li, X. Zhao, and H. Liang, Throughput maximization by deep reinforcement learning with energy cooperation for renewable ultradense IoT networks, IEEE Internet Things J., vol. 7, no. 9, pp. 9091–9102, 2020.
Crossref Google Scholar
[34]
D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, Deterministic policy gradient algorithms, in Proc. 31 st Int. Conf. Int. Conf. Machine Learning, Beijing, China, 2014, pp. I-387–I-395.
[35]

K. Li, Y. Emami, W. Ni, E. Tovar, and Z. Han, Onboard deep deterministic policy gradients for online flight resource allocation of UAVs, IEEE Netw. Lett., vol. 2, no. 3, pp. 106–110, 2020.
Crossref Google Scholar
[36]
H. Peng and X. S. Shen, DDPG-based resource management for MEC/UAV-assisted vehicular networks, in Proc. 2020 IEEE 92 nd Vehicular Technology Conf. (VTC2020-Fall ), Victoria, Canada, 2020, pp. 1–6.
Crossref
[37]

K. K. Nguyen, S. R. Khosravirad, D. B. Da Costa, L. D. Nguyen, and T. Q. Duong, Reconfigurable intelligent surface-assisted multi-UAV networks: Efficient resource allocation with deep reinforcement learning, IEEE J. Sel. Top. Signal Process., vol. 16, no. 3, pp. 358–368, 2022.
Crossref Google Scholar
[38]

J. Cui, Y. Liu, and A. Nallanathan, Multi-agent reinforcement learning-based resource allocation for UAV networks, IEEE Trans. Wirel. Commun., vol. 19, no. 2, pp. 729–743, 2020.
Crossref Google Scholar
[39]
M. Chen, W. Saad, and C. Yin, Liquid state machine learning for resource allocation in a network of cache-enabled LTE-U UAVs, in Proc. GLOBECOM 2017-2017 IEEE Global Communications Conf., Singapore, 2017, pp. 1–6.
Crossref
[40]

A. M. Seid, A. Erbad, H. N. Abishu, A. Albaseer, M. Abdallah, and M. Guizani, Multiagent federated reinforcement learning for resource allocation in UAV-enabled internet of medical things networks, IEEE Internet Things J., vol. 10, no. 22, pp. 19695–19711, 2023.
Crossref Google Scholar
[41]

J. Li, C. Zhou, J. Liu, M. Sheng, N. Zhao, and Y. Su, Reinforcement learning-based resource allocation for coverage continuity in high dynamic UAV communication networks, IEEE Trans. Wirel. Commun., vol. 23, no. 2, pp. 848–860, 2024.
Crossref Google Scholar
[42]
Y. Du, K. Wang, K. Yang, and G. Zhang, Energy-efficient resource allocation in UAV based MEC system for IoT devices, in Proc. 2018 IEEE Global Communications Conf. (GLOBECOM ), Abu Dhabi, United Arab Emirates, 2018, pp. 1–6.
Crossref
[43]

H. Peng and X. Shen, Multi-agent reinforcement learning based resource management in MEC- and UAV-assisted vehicular networks, IEEE J. Sel. Areas Commun., vol. 39, no. 1, pp. 131–141, 2021.
Crossref Google Scholar
[44]

Z. Wang, H. Rong, H. Jiang, Z. Xiao, and F. Zeng, A load-balanced and energy-efficient navigation scheme for UAV-mounted mobile edge computing, IEEE Trans. Netw. Sci. Eng., vol. 9, no. 5, pp. 3659–3674, 2022.
Crossref Google Scholar
[45]

Y. Yu, J. Tang, J. Huang, X. Zhang, D. K. C. So, and K. K. Wong, Multi-objective optimization for UAV-assisted wireless powered IoT networks based on extended DDPG algorithm, IEEE Trans. Commun., vol. 69, no. 9, pp. 6361–6374, 2021.
Crossref Google Scholar
[46]
H. Ju and B. Shim, Energy-efficient multi-UAV network using multi-agent deep reinforcement learning, in Proc. 2022 IEEE VTS Asia Pacific Wireless Communications Symp. (APWCS ), Seoul, Republic of Korea, 2022, pp. 70–74.
Crossref
[47]

T. Liu, T. Zhang, J. Loo, and Y. Wang, Deep reinforcement learning-based resource allocation for UAV-enabled federated edge learning, J. Commun. Inf. Netw., vol. 8, no. 1, pp. 1–12, 2023.
Crossref Google Scholar
Tsinghua Science and Technology
Volume 30 Issue 1,
February 2025
Pages 418-432
DOI: 10.26599/TST.2024.9010071
Cite this article:
Ahmad S, Zhang J, Nauman A, et al. Deep-EERA: DRL-Based Energy-Efficient Resource Allocation in UAV-Empowered Beyond 5G Networks. Tsinghua Science and Technology, 2025, 30(1): 418-432. https://doi.org/10.26599/TST.2024.9010071

845

Views

239

Downloads

0

Crossref

0

Web of Science

0

Scopus

0

CSCD

Altmetrics
Received: 05 December 2023
Revised: 16 February 2024
Accepted: 08 April 2024
Published: 11 September 2024
© The Author(s) 2025.

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).
Return