AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Complex System Modeling and Simulation Article
PDF (6.9 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Open Access

Dynamic Scheduling Algorithm Based on Evolutionary Reinforcement Learning for Sudden Contaminant Events Under Uncertain Environment

Chengyu Hu1, Rui Qiao1, Zhe Zhang1, Xuesong Yan1, Ming Li2
School of Computer Science, China University of Geosciences, Wuhan 430074, China
Department of Computer Science, California State University, Fresno, CA 93740, USA
Show Author Information

Abstract

For sudden drinking water pollution event, reasonable opening or closing valves and hydrants in a water distribution network (WDN), which ensures the isolation and discharge of contaminant as soon as possible, is considered as an effective emergency measure. In this paper, we propose an emergency scheduling algorithm based on evolutionary reinforcement learning (ERL), which can train a good scheduling policy by the combination of the evolutionary computation (EC) and reinforcement learning (RL). Then, the optimal scheduling policy can guide the operation of valves and hydrants in real time based on sensor information, and protect people from the risk of contaminated water. Experiments verify our algorithm can achieve good results and effectively reduce the impact of pollution events.

Keywords

scheduling problemevolutionary reinforcement learningwater distribution network

References

1

D. Hou, X. Ge, P. Huang, G. Zhang, and H. Loáiciga, A real-time, dynamic early-warning model based on uncertainty analysis and risk assessment for sudden water pollution accidents, Environmental Science and Pollution Research, vol. 21, no. 14, pp. 8878–8892, 2014.
Crossref Google Scholar
2

Y. Rui, D. Shen, S. Khalid, Z. Yang, and J. Wang, GIS-based emergency response system for sudden water pollution accidents, Physics and Chemistry of the Earth,Parts A/B/C, vol. 79, pp. 115–121, 2015.
Crossref Google Scholar
3

S. Rathi and R. Gupta, A simple sensor placement approach for regular monitoring and contamination detection in water distribution networks, KSCE Journal of Civil Engineering, vol. 20, no. 2, pp. 597–608, 2016.
Crossref Google Scholar
4

C. Hu, G. Ren, C. Liu, M. Li, and W. Jie, A spark-based genetic algorithm for sensor placement in large scale drinking water distribution systems, Cluster Computing, vol. 20, no. 2, pp. 1089–1099, 2017.
Crossref Google Scholar
5

X. Yan, K. Yang, C. Hu, and W. Gong, Pollution source positioning in a water supply network based on expensive optimization, Desalination and Water Treatment, vol. 110, pp. 308–318, 2018.
Crossref Google Scholar
6

D. Hou, X. Song, G. Zhang, H. Zhang, and H. Loaiciga, An early warning and control system for urban, drinking water quality protection: China’s experience, Environmental Science and Pollution Research, vol. 20, no. 7, pp. 4496–4508, 2013.
Crossref Google Scholar
7

H. Che, S. Liu, and K. Smith, Performance evaluation for a contamination detection method using multiple water quality sensors in an early warning system, Water, vol. 7, no. 4, pp. 1422–1436, 2015.
Crossref Google Scholar
8

T. Ren, S. Li, X. Zhang, and L. Liu, Maximum and minimum solutions for a nonlocal p-Laplacian fractional differential system from eco-economical processes, Boundary Value Problems, no. 1, p. 118, 2017.
Crossref Google Scholar
9

T. Ren, H. Xiao, Z. Zhou, X. Zhang, L. Xing, Z. Wang, and Y. Cui, The iterative scheme and the convergence analysis of unique solution for a singular fractional differential equation from the eco-economic complex system’s co-evolution process, Complexity, no. 3, pp. 1–15, 2019.
Crossref Google Scholar
10

A. Preis and A. Ostfeld, Multiobjective contaminant response modeling for water distribution systems security, Journal of Hydroinformatics, vol. 10, no. 4, pp. 267–274, 2008.
Crossref Google Scholar
11

A. Afshar and E. Najafi, Consequence management of chemical intrusion in water distribution networks under inexact scenarios, Journal of Hydroinformatics, vol. 16, no. 1, pp. 178–188, 2014.
Crossref Google Scholar
12

L. Wang, Z. Pan, and J. Wang, A review of reinforcement learning based intelligent optimization for manufacturing scheduling, Complex System Modeling and Simulation, vol. 1, no. 4, pp. 257–270, 2021.
Crossref Google Scholar
13

L. Luo, N. Zhao, and G. Lodewijks, Scheduling storage process of shuttle-based storage and retrieval systems based on reinforcement learning, Complex System Modeling and Simulation, vol. 1, no. 2, pp. 131–144, 2021.
Crossref Google Scholar
14
M. Gavanelli, M. Nonato, A. Peano, S. Alvisi, and M. Franchini, Genetic algorithms for scheduling devices operation in a water distribution system in response to contamination events, in Proc. 12th European Conference on Evolutionary Computation in Combinatorial Optimization, Málaga, Spain, 2012, pp. 124–135.
Crossref
15

A. Rasekh and K. Brumbelow, A dynamic simulation–optimization model for adaptive management of urban water distribution system contamination threats, Applied Soft Computing, vol. 32, pp. 59–71, 2015.
Crossref Google Scholar
16

M. E. Shafiee and E. Z. Berglund, Real-time guidance for hydrant flushing using sensor-hydrant decision trees, Journal of Water Resources Planning and Management, vol. 141, no. 6, pp. 04014079–1-04014079-14, 2015.
Crossref Google Scholar
17

M. E. Shafiee and E. Z. Berglund, Complex adaptive systems framework to simulate the performance of hydrant flushing rules and broadcasts during a water distribution system contamination event, Journal of Water Resources Planning and Management, vol. 143, no. 4, p. 04017001, 2017.
Crossref Google Scholar
18

R. Schwartz, M. Housh, and A. Ostfeld, Limited multistage stochastic programming for water distribution systems optimal operation, Journal of Water Resources Planning and Management, vol. 142, no. 10, pp. 1–6, 2016.
Crossref Google Scholar
19

P. Khatavkar and L. W. Mays, Optimization-simulation model for real-time pump and valve operation of water distribution systems under critical conditions, Urban Water Journal, vol. 16, no. 1, pp. 45–55, 2019.
Crossref Google Scholar
20

C. Hu, J. Cai, D. Zeng, X. Yan, W. Gong, and L. Wang, Deep reinforcement learning based valve scheduling for pollution isolation in water distribution network, Mathematical Biosciences and Engineering, vol. 17, no. 1, pp. 105–121, 2020.
Crossref Google Scholar
21

W. Gong, Z. Liao, X. Mi, L. Wang, and Y. Guo, Nonlinear equations solving with intelligent optimization algorithms: A survey, Complex System Modeling and Simulation, vol. 1, no. 1, pp. 15–32, 2021.
Crossref Google Scholar
22
S. Khadka, S. Majumdar, T. E. Nassar, Z. Dwiel, E. Tumer, S. Miret, Y. Liu, and K. Tumer, Collaborative evolutionary reinforcement learning, in Proc. 36th International Conference on Machine Learning, Long Beach, CA, USA, 2019, pp. 3341–3350.
23

S. Lü, S. Han, W. Zhou, and J. Zhang, Recruitment-imitation mechanism for evolutionary reinforcement learning, Information Sciences, vol. 553, pp. 172–188, 2021.
Crossref Google Scholar
24

A. Gupta, S. Savarese, S. Ganguli, and F. -F. Li, Embodied intelligence via learning and evolution, Nature Communications, vol. 12, pp. 1–12, 2021.
Crossref Google Scholar
25

B. Wu, H. -J. Jiang, C. Wang, and M. Dong, Knowledge and behavior-driven fruit fly optimization algorithm for field service scheduling problem with customer satisfaction, Complexity, no. 11, pp. 1–14, 2021.
Crossref Google Scholar
26

B. Wu, T. Yuan, Y. Qi, and M. Dong, Public opinion dissemination with incomplete information on social network: A study based on the infectious diseases model and game theory, Complex System Modeling and Simulation, vol. 1, no. 2, pp. 109–121, 2021.
Crossref Google Scholar
27

X. Han, Y. Han, Q. Chen, J. Li, H. Sang, Y. Liu, Q. Pan, and Y. Nojima, Distributed flow shop scheduling with sequence-dependent setup times using an improved iterated greedy algorithm, Complex System Modeling and Simulation, vol. 1, no. 3, pp. 198–217, 2021.
Crossref Google Scholar
28

W. Zhang, W. Hou, C. Li, W. Yang, and M. Gen, Multidirection update-based multiobjective particle swarm optimization for mixed no-idle flow-shop scheduling problem, Complex System Modeling and Simulation, vol. 1, no. 3, pp. 176–197, 2021.
Crossref Google Scholar
29

X. Wu, Z. Cao, and S. Wu, Real-time hybrid flow shop scheduling approach in smart manufacturing environment, Complex System Modeling and Simulation, vol. 1, no. 4, pp. 335–350, 2021.
Crossref Google Scholar
30
Y. Wei, X. Jiang, P. Hao, and K. Gu, Multi-agent co-evolutionary scheduling approach based on genetic reinforcement learning, in Proc. 2009 Fifth International Conference on Natural Computation, Tianjian, China, 2009, pp. 573–577.
31
D. Zeng, J. Zhan, W. Peng, and Z. Zeng, Evolutionary job scheduling with optimized population by deep reinforcement learning, Engineering Optimization, doi: 10.1080/0305215X.2021.2013479.
Crossref
32

A. Kamiya, H. Kimura, M. Yamamura, and S. Kobayashi, Power plant start-up scheduling: A reinforcement learning approach combined with evolutionary computation, Journal of Intelligent &Fuzzy Systems, vol. 6, no. 1, pp. 99–115, 1998.
Google Scholar
33

J. Dou, C. Chen, and P. Yang, Genetic scheduling and reinforcement learning in multirobot systems for intelligent warehouses, Mathematical Problems in Engineering, vol. 2015, pp. 1–10, 2015.
Crossref Google Scholar
34

E. K. Ahmed, Z. Li, B. Veeravalli, and S. Ren, Reinforcement learning-enabled genetic algorithm for school bus scheduling, Journal of Intelligent Transportation Systems, vol. 26, no. 3, pp. 269–283, 2020.
Crossref Google Scholar
35

Z. Xu, G. Han, L. Liu, M. Martínez-García, and Z. Wang, Multi-energy scheduling of an industrial integrated energy system by reinforcement learning-based differential evolution, IEEE Transactions on Green Communications and Networking, vol. 5, no. 3, pp. 1077–1090, 2021.
Crossref Google Scholar
36

Z. Li, X. Wei, X. Jiang, and Y. Pang, A kind of reinforcement learning to improve genetic algorithm for multiagent task scheduling, Mathematical Problems in Engineering, nos. 43–45, pp. 1–12, 2021.
Crossref Google Scholar
37
L. A. Rossman, Epanet 2: Users manual, https://www.epa.gov/water-research/epanet, 2022.
38
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT press, 2018.
39
M. V. Otterlo and M. Wiering, Reinforcement learning and Markov decision processes, in Reinforcement Learning, M. Wiering and M. Otterlo, eds. Berlin, Germany: Springer, 2012, pp. 3–42.
Crossref
40
D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv: 1412.6980, 2014.
Complex System Modeling and Simulation
Volume 2 Issue 3,
September 2022
Pages 213-223
DOI: 10.23919/CSMS.2022.0014
Cite this article:
Hu C, Qiao R, Zhang Z, et al. Dynamic Scheduling Algorithm Based on Evolutionary Reinforcement Learning for Sudden Contaminant Events Under Uncertain Environment. Complex System Modeling and Simulation, 2022, 2(3): 213-223. https://doi.org/10.23919/CSMS.2022.0014

698

Views

45

Downloads

5

Crossref

5

Scopus

Altmetrics
Received: 17 April 2022
Revised: 15 June 2022
Accepted: 15 July 2022
Published: 30 September 2022
© The author(s) 2022

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).
Return