The rise of innovative applications, like online gaming, smart healthcare, and Internet of Things (IoT) services, has increased demand for high data rates and seamless connectivity, posing challenges for Beyond 5G (B5G) networks. There is a need for cost-effective solutions to enhance spectral efficiency in densely populated areas, ensuring higher data rates and uninterrupted connectivity while minimizing costs. Unmanned Aerial Vehicles (UAVs) as Aerial Base Stations (ABSs) offer a promising and cost-effective solution to boost network capacity, especially during emergencies and high-data-rate demands. Nevertheless, integrating UAVs into the B5G networks presents new challenges, including resource scarcity, energy efficiency, resource allocation, optimal power transmission control, and maximizing overall throughput. This paper presents a UAV-assisted B5G communication system where UAVs act as ABSs, and introduces the Deep Reinforcement Learning (DRL) based Energy Efficient Resource Allocation (Deep-EERA) mechanism. An efficient DRL-based Deep Deterministic Policy Gradient (DDPG) mechanism is introduced for optimal resource allocation with the twin goals of energy efficiency and average throughput maximization. The proposed Deep-EERA method learns optimal policies to conserve energy and enhance throughput within the dynamic and complex UAV-empowered B5G environment. Through extensive simulations, we validate the performance of the proposed approach, demonstrating that it outperforms other baseline methods in energy efficiency and throughput maximization.