AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Communications in Transportation Research Article
Article Link
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Research Article | Open Access

Traffic expertise meets residual RL: Knowledge-informed model-based residual reinforcement learning for CAV trajectory control

Zihao Sheng, Zilin Huang, Sikai Chen( )
Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, WI 53706, USA
Show Author Information

Abstract

Model-based reinforcement learning (RL) is anticipated to exhibit higher sample efficiency than model-free RL by utilizing a virtual environment model. However, obtaining sufficiently accurate representations of environmental dynamics is challenging because of uncertainties in complex systems and environments. An inaccurate environment model may degrade the sample efficiency and performance of model-based RL. Furthermore, while model-based RL can improve sample efficiency, it often still requires substantial training time to learn from scratch, potentially limiting its advantages over model-free approaches. To address these challenges, this paper introduces a knowledge-informed model-based residual reinforcement learning framework aimed at enhancing learning efficiency by infusing established expert knowledge into the learning process and avoiding the issue of beginning from zero. Our approach integrates traffic expert knowledge into a virtual environment model, employing the intelligent driver model (IDM) for basic dynamics and neural networks for residual dynamics, thus ensuring adaptability to complex scenarios. We propose a novel strategy that combines traditional control methods with residual RL, facilitating efficient learning and policy optimization without the need to learn from scratch. The proposed approach is applied to connected automated vehicle (CAV) trajectory control tasks for the dissipation of stop-and-go waves in mixed traffic flows. The experimental results demonstrate that our proposed approach enables the CAV agent to achieve superior performance in trajectory control compared with the baseline agents in terms of sample efficiency, traffic flow smoothness and traffic mobility.

Keywords

Model-based reinforcement learningResidual policy learningMixed traffic flowConnected automated vehicles

References

 
Agarwal, A., Kakade, S.M., Lee, J.D., Mahajan, G., 2020. Optimality and approximation with policy gradient methods in markov decision processes. In: Proceedings of Thirty Third Conference on Learning Theory, pp. 64–66.
 

Andreotti, E., Selpi, Boyraz, P., 2023. Potential impact of autonomous vehicles in mixed traffic from simulation using real traffic flow. J. Int. Con. Veh. 6, 1–15.
Crossref Google Scholar
 

Chee, K.Y., Jiahao, T.Z., Hsieh, M.A., 2022. KNODE-MPC: a knowledge-based data-driven predictive control framework for aerial robots. IEEE Rob. Autom. Lett. 7, 2819–2826.
Crossref Google Scholar
 

Chen, D., Hajidavalloo, M.R., Li, Z., Chen, K., Wang, Y., Jiang, L., et al., 2023a. Deep multi-agent reinforcement learning for highway on-ramp merging in mixed traffic. IEEE Trans. Intell. Transport. Syst. 24, 11623–11638.
Crossref Google Scholar
 

Chen, S., Dong, J., Ha, P.Y.J., Li, Y., Labi, S., 2021. Graph neural network and reinforcement learning for multi-agent cooperative control of connected autonomous vehicles. Computer Aided Civil Eng. 36, 838–857.
Crossref Google Scholar
 

Chen, S., Zong, S., Chen, T., Huang, Z., Chen, Y., Labi, S., 2023b. A taxonomy for autonomous vehicles considering ambient road infrastructure. Sustainability 15, 11258.
Crossref Google Scholar
 

Chua, K., Calandra, R., McAllister, R., Levine, S., 2018. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Adv. Neural Inf. Process. Syst 31.
Google Scholar
 
Cui, S., Seibold, B., Stern, R., Work, D.B., 2017. Stabilizing traffic flow via a single autonomous vehicle: possibilities and limitations. In: 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 1336–1341.
Crossref
 

Ding, H., Li, W., Xu, N., Zhang, J., 2022. An enhanced eco-driving strategy based on reinforcement learning for connected electric vehicles: cooperative velocity and lane-changing control. J. Intell. Connect. Veh. 5, 316–332.
Crossref Google Scholar
 
Dong, J., Chen, S., Labi, S., 2021. Addressing crash-imminent situations caused by human driven vehicle errors in a mixed traffic stream: a model-based reinforcement learning approach for CAV. https://arxiv.org/abs/2110.05556v1.
 

Dong, J., Chen, S., Miralinaghi, M., Chen, T., Labi, S., 2022. Development and testing of an image transformer for explainable autonomous driving systems. J. Intell. Connect. Veh. 5, 235–249.
Crossref Google Scholar
 

Dong, J., Chen, S., Miralinaghi, M., Chen, T., Li, P., Labi, S., 2023. Why did the AI make that decision? Towards an explainable artificial intelligence (XAI) for autonomous driving systems. Transport. Res. C Emerg. Technol. 156, 104358.
Crossref Google Scholar
 

Du, R., Chen, S., Dong, J., Chen, T., Fu, X., Labi, S., 2024. Dynamic urban traffic rerouting with fog-cloud reinforcement learning. Computer Aided Civil Eng. 39, 793–813.
Crossref Google Scholar
 

Feng, S., Song, Z., Li, Z., Zhang, Y., Li, L., 2021. Robust platoon control in mixed traffic flow based on tube model predictive control. IEEE Trans. Intell. Veh. 6, 711–722.
Crossref Google Scholar
 

Garriga, J.L., Soroush, M., 2010. Model predictive control tuning methods: a review. Ind. Eng. Chem. Res. 49, 3505–3515.
Crossref Google Scholar
 

Gong, S., Du, L., 2018. Cooperative platoon control for a mixed traffic flow including human drive vehicles and connected and autonomous vehicles. Transp. Res. Part B Methodol. 116, 25–61.
Crossref Google Scholar
 

Guo, J., Cheng, L., Wang, S., 2023. CoTV: cooperative control for traffic light signals and connected autonomous vehicles using deep reinforcement learning. IEEE Trans. Intell. Transport. Syst. 24, 10501–10512.
Crossref Google Scholar
 

Ha, P.Y.J., Chen, S., Dong, J., Labi, S., 2023. Leveraging vehicle connectivity and autonomy for highway bottleneck congestion mitigation using reinforcement learning. Transp. A Transp. Sci. 1–26.
Crossref Google Scholar
 
Haarnoja, T., Zhou, A., Abbeel, P., Levine, S., 2018. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International Conference on Machine Learning, pp. 1861–1870.
 

Han, Y., Wang, M., Leclercq, L., 2023. Leveraging reinforcement learning for dynamic traffic control: a survey and challenges for field implementation. Commun. Transp. Res. 3, 100104.
Crossref Google Scholar
 

Han, Y., Wang, M., Li, L., Roncoli, C., Gao, J., Liu, P., 2022. A physics-informed reinforcement learning-based strategy for local and coordinated ramp metering. Transport. Res. C Emerg. Technol. 137, 103584.
Crossref Google Scholar
 

Hornik, K., Stinchcombe, M., White, H., 1989. Multilayer feedforward networks are universal approximators. Neural Network. 2, 359–366.
Crossref Google Scholar
 

Hou, J., Chen, G., Li, Z., He, W., Gu, S., Knoll, A., et al., 2024. Hybrid residual multiexpert reinforcement learning for spatial scheduling of high-density parking lots. IEEE Trans. Cybern. 54, 2771–2783.
Crossref Google Scholar
 

Hou, X., Gan, M., Zhang, J., Zhao, S., Ji, Y., 2023. Vehicle ride comfort optimization in the post-braking phase using residual reinforcement learning. Adv. Eng. Inf. 58, 102198.
Crossref Google Scholar
 
Huang, X., Wu, D., Jenkin, M., Boulet, B., 2021. ModelLight: model-based metareinforcement learning for traffic signal control. https://arxiv.org/abs/2111.08067v2.
 

Huang, Z., Chen, S., Pian, Y., Sheng, Z., Ahn, S., Noyce, D.A., 2024a. Toward C-V2X enabled connected transportation system: RSU-based cooperative localization framework for autonomous vehicles. IEEE Trans. Intell. Transport. Syst. 1–15.
Crossref Google Scholar
 

Huang, Z., Sheng, Z., Ma, C., Chen, S., 2024b. Human as AI mentor: enhanced human-in-the-loop reinforcement learning for safe and efficient autonomous driving. Commun. Transp. Res. 4, 100127.
Crossref Google Scholar
 

Janner, M., Fu, J., Zhang, M., Levine, S., 2019. When to trust your model: model-based policy optimization. Adv. Neural Inf. Process. Syst. 32.
Google Scholar
 
Johannink, T., Bahl, S., Nair, A., Luo, J., Kumar, A., Loskyll, M., et al., 2019. Residual reinforcement learning for robot control. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 6023–6029.
Crossref
 

Kabzan, J., Hewing, L., Liniger, A., Zeilinger, M.N., 2019. Learning-based model predictive control for autonomous racing. IEEE Rob. Autom. Lett. 4, 3363–3370.
Crossref Google Scholar
 

Karniadakis, G.E., Kevrekidis, I.G., Lu, L., Perdikaris, P., Wang, S., Yang, L., 2021. Physics-informed machine learning. Nat. Rev. Phys. 3, 422–440.
Crossref Google Scholar
 

Kiran, B.R., Sobh, I., Talpaert, V., Mannion, P., Al Sallab, A.A., Yogamani, S., et al., 2022. Deep reinforcement learning for autonomous driving: a survey. IEEE Trans. Intell. Transport. Syst. 23, 4909–4926.
Crossref Google Scholar
 

Lee, H., Kim, N., Cha, S.W., 2020. Model-based reinforcement learning for eco-driving control of electric vehicles. IEEE Access 8, 202886–202896.
Crossref Google Scholar
 

Li, J., Yu, C., Shen, Z., Su, Z., Ma, W., 2023. A survey on urban traffic control under mixed traffic environment with connected automated vehicles. Transport. Res. C Emerg. Technol. 154, 104258.
Crossref Google Scholar
 

Liao, H., Shen, H., Li, Z., Wang, C., Li, G., Bie, Y., et al., 2024. GPT-4 enhanced multimodal grounding for autonomous driving: leveraging cross-modal attention with large language models. Commun. Transp. Res. 4, 100116.
Crossref Google Scholar
 

Lidstrom, K., Sjoberg, K., Holmberg, U., Andersson, J., Bergh, F., Bjade, M., et al., 2012. A modular CACC system integration and design. IEEE Trans. Intell. Transport. Syst.13, 1050–1061.
Crossref Google Scholar
 

Liu, C., Sheng, Z., Chen, S., Shi, H., Ran, B., 2023a. Longitudinal control of connected and automated vehicles among signalized intersections in mixed traffic flow with deep reinforcement learning approach. Phys. Stat. Mech. Appl. 629, 129189.
Crossref Google Scholar
 

Liu, C., Sheng, Z., Li, P., Chen, S., Luo, X., Ran, B., 2024. A distributed deep reinforcement learning-based longitudinal control strategy for connected automated vehicles combining attention mechanism. Transp. Lett. 1–17.
Crossref Google Scholar
 

Liu, Y., Wu, F., Liu, Z., Wang, K., Wang, F., Qu, X., 2023b. Can language models be used for real-world urban-delivery route optimization? Innovation 4, 100520.
Crossref Google Scholar
 
Long, K., Sheng, Z., Shi, H., Li, X., Chen, S., Ahn, S., 2023. A physics enhanced residual learning (PERL) framework for vehicle trajectory prediction. https://arxiv.org/abs/2309.15284v2.
 
Lopez, P.A., Wiessner, E., Behrisch, M., Bieker-Walz, L., Erdmann, J., Flotterod, Y.P., et al., 2018. Microscopic traffic simulation using SUMO. In: 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 2575–2582.
Crossref
 

Milanes, V., Shladover, S.E., Spring, J., Nowakowski, C., Kawazoe, H., Nakamura, M., 2014. Cooperative adaptive cruise control in real traffic situations. IEEE Trans. Intell. Transport. Syst. 15, 296–305.
Crossref Google Scholar
 

Mo, Z., Shi, R., Di, X., 2021. A physics-informed deep learning paradigm for car-following models. Transport. Res. C Emerg. Technol. 130, 103240.
Crossref Google Scholar
 

Moerland, T.M., Broekens, J., Plaat, A., Jonker, C.M., 2023. Model-based reinforcement learning: a survey. FNT. Mach. Learn. 16, 1–118.
Crossref Google Scholar
 

Newell, G.F., 2002. A simplified car-following theory: a lower order model. Transp. Res. Part B Methodol. 36, 195–205.
Crossref Google Scholar
 

O'Connell, M., Shi, G., Shi, X., Azizzadenesheli, K., Anandkumar, A., Yue, Y., et al., 2022. Neural-Fly enables rapid learning for agile flight in strong winds. Sci. Robot. 7, 195–205.
Crossref Google Scholar
 

Olovsson, T., Svensson, T., Wu, J., 2022. Future connected vehicles: communications demands, privacy and cyber-security. Commun. Transp. Res. 2, 100056.
Crossref Google Scholar
 

Pan, T., Guo, R., Lam, W.H.K., Zhong, R., Wang, W., He, B., 2021. Integrated optimal control strategies for freeway traffic mixed with connected automated vehicles: a model-based reinforcement learning approach. Transport. Res. C Emerg. Technol. 123, 102987.
Crossref Google Scholar
 

Peng, B., Keskin, M.F., Kulcsár, B., Wymeersch, H., 2021. Connected autonomous vehicles for improving mixed traffic efficiency in unsignalized intersections with deep reinforcement learning. Commun. Transp. Res. 1, 100017.
Crossref Google Scholar
 

Qu, X., Lin, H., Liu, Y., 2023. Envisioning the future of transportation: inspiration of ChatGPT and large models. Commun. Transp. Res. 3, 100103.
Crossref Google Scholar
 
Schulman, J., Levine, S., Abbeel, P., Jordan, M., Moritz, P., 2015. Trust region policy optimization. In: International Conference on Machine Learning, pp. 1889–1897.
 
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O., 2017. Proximal policy optimization algorithms. https://arxiv.org/abs/1707.06347v2.
 

Sheng, Z., Huang, Z., Chen, S., 2024a. Kinematics-aware multigraph attention network with residual learning for heterogeneous trajectory prediction. J. Intell. Connect. Veh. 7, 138–150.
Crossref Google Scholar
 
Sheng, Z., Huang, Z., Chen, S., 2024b. Ego-planning-guided multi-graph convolutional network for heterogeneous agent trajectory prediction. https://doi.org/10.1111/mice.13301.
Crossref
 

Sheng, Z., Liu, L., Xue, S., Zhao, D., Jiang, M., Li, D., 2023. A cooperation-aware lane change method for automated vehicles. IEEE Trans. Intell. Transport. Syst. 24, 3236–3251.
Crossref Google Scholar
 

Shi, H., Chen, D., Zheng, N., Wang, X., Zhou, Y., Ran, B., 2023. A deep reinforcement learning based distributed control strategy for connected automated vehicles in mixed traffic platoon. Transport. Res. C Emerg. Technol. 148, 104019.
Crossref Google Scholar
 

Shi, R., Mo, Z., Huang, K., Di, X., Du, Q., 2022. A physics-informed deep learning paradigm for traffic state and fundamental diagram estimation. IEEE Trans. Intell. Transport. Syst. 23, 11688–11698.
Crossref Google Scholar
 

Staessens, T., Lefebvre, T., Crevecoeur, G., 2022. Adaptive control of a mechatronic system using constrained residual reinforcement learning. IEEE Trans. Ind. Electron. 69, 10447–10456.
Crossref Google Scholar
 

Stern, R.E., Cui, S., Delle Monache, M.L., Bhadani, R., Bunting, M., Churchill, M., et al., 2018. Dissipation of stop-and-go waves via control of autonomous vehicles: field experiments. Transport. Res. C Emerg. Technol. 89, 205–221.
Crossref Google Scholar
 
Sun, Z., Huang, J., Ji, A., Zhao, R., Zheng, G., 2023. Cooperative merging for connected automated vehicles in mixed traffic: a multi-agent reinforcement learning approach. https://doi.org/10.2139/ssrn.4564695.
Crossref
 
Sutton, R.S., 1990. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In: Machine Learning Proceedings 1990, pp. 216–224.
Crossref
 

Treiber, M., Hennecke, A., Helbing, D., 2000. Congested traffic states in empirical observations and microscopic simulations. Phys. Rev. E 62, 1805–1824.
Crossref Google Scholar
 

Vahidi, A., Eskandarian, A., 2003. Research advances in intelligent collision avoidance and adaptive cruise control. IEEE Trans. Intell. Transport. Syst. 4, 143–153.
Crossref Google Scholar
 

Wang, S., Li, Z., Wang, B., Li, M., 2024. Collision avoidance motion planning for connected and automated vehicle platoon merging and splitting with a hybrid automaton architecture. IEEE Trans. Intell. Transport. Syst. 25, 1445–1464.
Crossref Google Scholar
 

Wang, W., Zhang, Y., Gao, J., Jiang, Y., Yang, Y., Zheng, Z., et al., 2023. GOPS: a general optimal control problem solver for autonomous driving and industrial control applications. Commun. Transp. Res. 3, 100096.
Crossref Google Scholar
 

Wu, C., Kreidieh, A.R., Parvate, K., Vinitsky, E., Bayen, A.M., 2022. Flow: a modular learning framework for mixed autonomy traffic. IEEE Trans. Robot. 38, 1270–1286.
Crossref Google Scholar
 

Wu, J., Huang, Z., Lv, C., 2023. Uncertainty-aware model-based reinforcement learning: methodology and application in autonomous driving. IEEE Trans. Intell. Veh. 8, 194–203.
Crossref Google Scholar
 

Wu, J., Qu, X., 2022. Intersection control with connected and automated vehicles: a review. J. Intell. Connect. Veh. 5, 260–269.
Crossref Google Scholar
 

Yang, J., Zhao, D., Lan, J., Xue, S., Zhao, W., Tian, D., et al., 2023. Eco-driving of general mixed platoons with CAVs and HDVs. IEEE Trans. Intell. Veh. 8, 1190–1203.
Crossref Google Scholar
 

Yang, Z., Zheng, Z., Kim, J., Rakha, H., 2024. Eco-driving strategies using reinforcement learning for mixed traffic in the vicinity of signalized intersections. Transport. Res. C Emerg. Technol. 165, 104683.
Crossref Google Scholar
 
Yavas, U., Kumbasar, T., Ure, N.K., 2022. Model-based reinforcement learning for advanced adaptive cruise control: a hybrid car following policy. In: 2022 IEEE Intelligent Vehicles Symposium (IV), pp. 1466–1472.
Crossref
 

Yu, M., Long, J., 2022. An eco-driving strategy for partially connected automated vehicles at a signalized intersection. IEEE Trans. Intell. Transport. Syst. 23, 15780–15793.
Crossref Google Scholar
 

Yue, L., Abdel-Aty, M., Wang, Z., 2022. Effects of connected and autonomous vehicle merging behavior on mainline human-driven vehicle. J. Intell. Connect. Veh. 5, 36–45.
Crossref Google Scholar
 

Zhang, R., Hou, J., Chen, G., Li, Z., Chen, J., Knoll, A., 2022. Residual policy learning facilitates efficient model-free autonomous racing. IEEE Rob. Autom. Lett. 7, 11625–11632.
Crossref Google Scholar
 

Zheng, Y., Wang, J., Li, K., 2020. Smoothing traffic flow via control of autonomous vehicles. IEEE Internet Things J. 7, 3882–3896.
Crossref Google Scholar
 

Zhou, D., Gayah, V., 2023. Improving deep reinforcement learning-based perimeter metering control methods with domain control knowledge. Transport. Res. Rec. 2677, 384–405.
Crossref Google Scholar
 

Zhou, Y., Ahn, S., Wang, M., Hoogendoorn, S., 2020. Stabilizing mixed vehicular platoons with connected automated vehicles: an H-infinity approach. Transp. Res. Part B Methodol. 132, 152–170.
Crossref Google Scholar
 

Zhu, J., Easa, S., Gao, K., 2022. Merging control strategies of connected and autonomous vehicles at freeway on-ramps: a comprehensive review. J. Intell. Connect. Veh. 5,99–111.
Crossref Google Scholar
Communications in Transportation Research
Volume 4 Issue 4,
December 2024
Article number: 100142
DOI: 10.1016/j.commtr.2024.100142
Cite this article:
Sheng Z, Huang Z, Chen S. Traffic expertise meets residual RL: Knowledge-informed model-based residual reinforcement learning for CAV trajectory control. Communications in Transportation Research, 2024, 4(4): 100142. https://doi.org/10.1016/j.commtr.2024.100142

11

Views

1

Crossref

1

Web of Science

1

Scopus

Altmetrics
Received: 05 May 2024
Revised: 31 July 2024
Accepted: 07 August 2024
Published: 18 October 2024
© 2024 The Authors.

This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
Return