Enhanced reinforcement learning-model predictive control for distributed energy systems: Overcoming local and global optimization limitations
), Yingjun Ruan1()Abstract
The complex structures of distributed energy systems (DES) and uncertainties arising from renewable energy sources and user load variations pose significant operational challenges. Model predictive control (MPC) and reinforcement learning (RL) are widely used to optimize DES by predicting future outcomes based on the current state. However, MPC’s real-time application is constrained by its computational demands, making it less suitable for complex systems with extended predictive horizons. Meanwhile, RL’s model-free approach leads to suboptimal data utilization, limiting its overall performance. To address these issues, this study proposes an improved reinforcement learning-model predictive control (RL-MPC) algorithm that combines the high-precision local optimization of MPC with the global optimization capability of RL. In this study, we enhance the existing RL-MPC algorithm by increasing the number of optimization steps performed by the MPC component. We evaluated RL, MPC, and the enhanced RL-MPC on a DES comprising a photovoltaic (PV) and battery energy storage system (BESS). The results indicate the following: (1) The twin delayed deep deterministic policy gradient (TD3) algorithm outperforms other RL algorithms in energy cost optimization, but is outperformed in all cases by RL-MPC. (2) For both MPC and RL-MPC, when the mean absolute percentage error (MAPE) of the first-step prediction is 5%, the total cost increases by ~1.2% compared to that when the MAPE is 0%. However, if the accuracy of the initial prediction data remains constant while only the error gradient of the data sequence increases, the total cost remains nearly unchanged, with an increase of only ~0.1%. (3) Within a 12 h predictive horizon, RL-MPC outperforms MPC, suggesting it as a suitable alternative to MPC when high-accuracy prediction data are limited.
Keywords
References
Afram A, Janabi-Sharifi F (2014). Theory and applications of HVAC control systems—A review of model predictive control (MPC). Building and Environment, 72: 343–355.
Arroyo J, Spiessens F, Helsen L (2020). Identification of multi-zone grey-box building models for use in model predictive control. Journal of Building Performance Simulation, 13: 472–486.
Arroyo J, Manna C, Spiessens F, et al. (2022). Reinforced model predictive control (RL-MPC) for building energy management. Applied Energy, 309: 118346.
Bayendang NP, Kahn MT, Balyan V (2023). Combined cold, heat and power (CCHP) systems and fuel cells for CCHP applications: A topological review. Clean Energy, 7: 436–491.
Bui VH, Hussain A, Kim HM (2020). Double deep Q-learning-based distributed operation of battery energy storage system considering uncertainties. IEEE Transactions on Smart Grid, 11: 457–469.
Bürger A, Bohlayer M, Hoffmann S, et al. (2020). A whole-year simulation study on nonlinear mixed-integer model predictive control for a thermal energy supply system with multi-use components. Applied Energy, 258: 114064.
Cai W, Sawant S, Reinhardt D, et al. (2023). A learning-based model predictive control strategy for home energy management systems. IEEE Access, 11: 145264–145280.
Ceusters G, Rodríguez RC, García AB, et al. (2021). Model-predictive control and reinforcement learning in multi-energy system case studies. Applied Energy, 303: 117634.
Chen P, Liu M, Chen C, et al. (2019). A battery management strategy in microgrid for personalized customer requirements. Energy, 189: 116245.
Chen B, Cai Z, Bergés M (2020). Gnu-RL: A practical and scalable reinforcement learning solution for building HVAC control using a differentiable MPC policy. Frontiers in Built Environment, 6: 562239.
Dong X, Zhang C, Sun B (2022). Optimization strategy based on robust model predictive control for RES-CCHP system under multiple uncertainties. Applied Energy, 325: 119707.
Dreher A, Bexten T, Sieker T, et al. (2022). AI agents envisioning the future: Forecast-based operation of renewable energy storage systems using hydrogen with deep reinforcement learning. Energy Conversion and Management, 258: 115401.
Drgoňa J, Arroyo J, Cupeiro Figueroa I, et al. (2020). All you need to know about model predictive control for buildings. Annual Reviews in Control, 50: 190–232.
Drgoňa J, Tuor A, Skomski E, et al. (2021). Deep learning explicit differentiable predictive control laws for buildings. IFAC-PapersOnLine, 54: 14–19.
Du H, Shen P, Chai WS, et al. (2022). Perspective and analysis of ammonia-based distributed energy system (DES) for achieving low carbon community in China. iScience, 25: 105120.
Fu Q, Han Z, Chen J, et al. (2022). Applications of reinforcement learning for building energy efficiency control: A review. Journal of Building Engineering, 50: 104165.
Gamarra C, Guerrero JM (2015). Computational optimization techniques applied to microgrids planning: A review. Renewable and Sustainable Energy Reviews, 48: 413–424.
Gros S, Zanon M (2020a). Data-driven economic NMPC using reinforcement learning. IEEE Transactions on Automatic Control, 65: 636–648.
Gros S, Zanon M (2020b). Reinforcement learning for mixed-integer problems based on MPC. IFAC-PapersOnLine, 53: 5219–5224.
Guo C, Wang X, Zheng Y, et al. (2022). Real-time optimal energy management of microgrid with uncertainties based on deep reinforcement learning. Energy, 238: 121873.
Han J, Ouyang L, Xu Y, et al. (2016). Current status of distributed energy system in China. Renewable and Sustainable Energy Reviews, 55: 288–297.
Hewing L, Wabersich KP, Menner M, et al. (2020). Learning-based model predictive control: Toward safe learning in control. Annual Review of Control, Robotics, and Autonomous Systems, 3: 269–296.
Hua H, Qin Y, Hao C, et al. (2019). Optimal energy management strategies for energy Internet via deep reinforcement learning approach. Applied Energy, 239: 598–609.
Huo Y, Bouffard F, Joós G (2022). Integrating learning and explicit model predictive control for unit commitment in microgrids. Applied Energy, 306: 118026.
Ławryńczuk M, Marusak PM, Chaber P, et al. (2022). Initialisation of optimisation solvers for nonlinear model predictive control: Classical vs. hybrid methods. Energies, 15: 2483.
Liang Y, He Y, Niu Y (2022). Robust errorless-control-targeted technique based on MPC for microgrid with uncertain electric vehicle energy storage systems. Energies, 15: 1398.
Lin Y, McPhee J, Azad NL (2021). Comparison of deep reinforcement learning and model predictive control for adaptive cruise control. IEEE Transactions on Intelligent Vehicles, 6: 221–231.
Lin M, Sun Z, Xia Y, et al. (2024). Reinforcement learning-based model predictive control for discrete-time systems. IEEE Transactions on Neural Networks and Learning Systems, 35: 3312–3324.
Liu X, Feng L, Kong X (2022a). A comparative study of robust MPC and stochastic MPC of wind power generation system. Energies, 15: 4814.
Liu F, Mo Q, Yang Y, et al. (2022b). A nonlinear model-based dynamic optimal scheduling of a grid-connected integrated energy system. Energy, 243: 123115.
Mahmud K, Khan B, Ravishankar J, et al. (2020). An internet of energy framework with distributed energy resources, prosumers and small-scale virtual power plants: An overview. Renewable and Sustainable Energy Reviews, 127: 109840.
Negenborn RR, De Schutter B, Wiering MA, et al. (2005). Learning-based model predictive control for Markov decision processes IFAC proceedings volumes. IFAC Proceedings Volumes, 38: 354–359.
Park J, Jeon S, Han S (2024). Model-based reinforcement learning with probabilistic ensemble terminal critics for data-efficient control applications. IEEE Transactions on Industrial Electronics, 71: 9470–9479.
Perera ATD, Kamalaruban P (2021). Applications of reinforcement learning in energy systems. Renewable and Sustainable Energy Reviews, 137: 110618.
Pinto G, Deltetto D, Capozzoli A (2021). Data-driven district energy management with surrogate models and deep reinforcement learning. Applied Energy, 304: 117642.
Qiu D, Dong Z, Zhang X, et al. (2022). Safe reinforcement learning for real-time automatic control in a smart energy-hub. Applied Energy, 309: 118403.
Qiu Y, Zhou S, Xia D, et al. (2023). Local integrated energy system operational optimization considering multi-type uncertainties: A reinforcement learning approach based on improved TD3 algorithm. IET Renewable Power Generation, 17: 2236–2256.
Raffin A, Hill A, Gleave A, et al. (2021). Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22: 1–8.
Ruan Y, Liang Z, Qian F, et al. (2023). Operation strategy optimization of combined cooling, heating, and power systems with energy storage and renewable energy based on deep reinforcement learning. Journal of Building Engineering, 65: 105682.
Salimi M, Hosseinpour M, Mansouri S, et al. (2022). Environmental aspects of the combined cooling, heating, and power (CCHP) systems: A review. Processes, 10: 711.
Shang C, Fu L, Bao X, et al. (2022). Energy optimal dispatching of ship’s integrated power system based on deep reinforcement learning. Electric Power Systems Research, 208: 107885.
Sun D, Jamshidnejad A, De Schutter B (2024). A novel framework combining MPC and deep reinforcement learning with application to freeway traffic control. IEEE Transactions on Intelligent Transportation Systems, 25: 6756–6769.
Tumeran NL, Yusoff SH, Gunawan TS, et al. (2023). Model predictive control based energy management system literature assessment for RES integration. Energies, 16: 3362.
Wu J, He H, Peng J, et al. (2018). Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus. Applied Energy, 222: 799–811.
Yang T, Zhao L, Li W, et al. (2021). Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning. Energy, 235: 121377.
Yang S, Wan MP (2022). Machine-learning-based model predictive control with instantaneous linearization—A case study on an air-conditioning and mechanical ventilation system. Applied Energy, 306: 118041.
Yu L, Qin S, Zhang M, et al. (2021). A review of deep reinforcement learning for smart building energy management. IEEE Internet of Things Journal, 8: 12046–12063.
Zanon M, Gros S (2021). Safe reinforcement learning using robust MPC. IEEE Transactions on Automatic Control, 66: 3638–3652.
Zhang Y, Zhang T, Wang R, et al. (2015). Optimal operation of a smart residential microgrid based on model predictive control by considering uncertainties and storage impacts. Solar Energy, 122: 1052–1065.
Zhang B, Hu W, Cao D, et al. (2019). Deep reinforcement learning-based approach for optimizing energy conversion in integrated electrical and heating system with renewable energy. Energy Conversion and Management, 202: 112199.
Zhang B, Hu W, Li J, et al. (2020). Dynamic energy conversion and management strategy for an integrated electricity and natural gas system with renewable energy: Deep reinforcement learning approach. Energy Conversion and Management, 220: 113063.
Zhang G, Hu W, Cao D, et al. (2021a). Data-driven optimal energy management for a wind-solar-diesel-battery-reverse osmosis hybrid energy system using a deep reinforcement learning approach. Energy Conversion and Management, 227: 113608.
Zhang B, Hu W, Cao D, et al. (2021b). Soft actor-critic-based multi-objective optimized energy conversion and management strategy for integrated energy systems with renewable energy. Energy Conversion and Management, 243: 114381.
Zhang Y, Cheng C, Cai H, et al. (2022). Long-term stochastic model predictive control and efficiency assessment for hydro-wind-solar renewable energy supply system. Applied Energy, 316: 119134.
京公网安备11010802044758号