AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home CSEE Journal of Power and Energy Systems Article
PDF (1.4 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Open Access

Multi-agent Simulation for Strategic Bidding in Electricity Markets Using Reinforcement Learning

Jidong Wang, Jiahui Wu, Xiangyu Kong ( )
Key Laboratory of Smart Grid of Ministry of Education, Tianjin University, Tianjin 300072, China
Show Author Information

Abstract

In this paper, a theoretical framework of Multi-agent Simulation (MAS) is proposed for strategic bidding in electricity markets using reinforcement learning, which consists of two parts: one is a MAS system used to simulate the competitive bidding of the actual electricity market; the other is an adaptive learning strategy bidding system used to provide agents with more intelligent bidding strategies. An Experience-Weighted Attraction (EWA) reinforcement learning algorithm (RLA) is applied to the MAS model and a new MAS method is presented for strategic bidding in electricity markets using a new Improved EWA (IEWA). From both qualitative and quantitative perspectives, it is compared with three other MAS methods using the Roth-Erev (RE), Q-learning and EWA. The results show that the performance of the MAS method using IEWA is proved to be better than the others. The four MAS models using four RLAs are built for strategic bidding in electricity markets. Through running the four MAS models, the rationality and correctness of the four MAS methods are verified for strategic bidding in electricity markets using reinforcement learning.

Keywords

reinforcement learningstrategic biddingElectricity marketmulti-agent simulation

References

[1]

M. Prabavathi and R. Gnanadass, "Electric power bidding model for practical utility system," Alexandria Engineering Journal, vol. 57, no. 1, pp. 277–286, Mar. 2018.
Crossref Google Scholar
[2]

C. Silva, B. F. Wollenberg, and C. Z. Zheng, "Application of mechanism design to electric power markets," IEEE Transactions on Power Systems, vol. 16, no. 1, pp. 1–8, Feb. 2001.
Crossref Google Scholar
[3]

F. J. Nogales, J. Contreras, A. J. Conejo, and R. Espínola, "Forecasting next-day electricity prices by time series models," IEEE Transactions on Power Systems, vol. 17, no. 2, pp. 342–348, May. 2002.
Crossref Google Scholar
[4]

D. L. Xiao and Q. Wei, "Hybrid Scenario Generation Method for Stochastic Virtual Bidding in Electricity Market," CSEE Journal of Power and Energy Systems, pp. 1312–1321, vol. 7, no. 6, Nov. 2021.
Google Scholar
[5]

D. S. Kirschen, "Demand-side view of electricity markets," IEEE Transactions on Power Systems, vol. 18, no. 2, pp. 520–527, May. 2003.
Crossref Google Scholar
[6]

C. L. Su and D. Kirschen, "Quantifying the effect of demand response on electricity markets," IEEE Transactions on Power Systems, vol. 24, no. 3, pp. 1199–1207, Aug. 2009.
Crossref Google Scholar
[7]

T. Ding, Z. Y. Wu, J. J. Lv, Z. H. Bie, and X. J. Zhang, "Robust co-optimization to energy and ancillary service joint dispatch considering wind power uncertainties in real-time electricity markets," IEEE Transactions on Sustainable Energy, vol. 7, no. 4, pp. 1547–1557, Oct. 2016.
Crossref Google Scholar
[8]

W. Jia, T. Ding, C. Huang, Z. K. Wang, Q. Zhou, and M. Shahidehpour, "Convex optimization of integrated power-gas energy flow model with applications to probabilistic energy flow," IEEE Transactions on Power Systems, vol. 36, no. 2, pp. 1432–1441, Mar. 2021.
Crossref Google Scholar
[9]

T. Ding, Z. K. Wang, W. H. Jia, B. Chen, C. Chen, and M. Shahidehpour, "Multiperiod distribution system restoration with routing repair crews, mobile electric vehicles, and soft-open-point networked microgrids," IEEE Transactions on Smart Grid, vol. 11, no. 6, pp. 4795–4808, Nov. 2020.
Crossref Google Scholar
[10]

A. Weidlich and D. Veit, "A critical survey of agent-based wholesale electricity market models," Energy Economics, vol. 30, no. 4, pp. 1728–1759, Jul. 2008.
Crossref Google Scholar
[11]

J. H. Qin, Q. C. Ma, Y. Shi, and L. Wang, "Recent advances in consensus of multi-agent systems: a brief survey," IEEE Transactions on Industrial Electronics, vol. 64, no. 6, pp. 4972–4983, Jun. 2017.
Crossref Google Scholar
[12]

P. Ringler, D. Keles, and W. Fichtner, "Agent-based modelling and simulation of smart electricity grids and markets – A literature review," Renewable and Sustainable Energy Reviews, vol. 57, pp. 205–215, May. 2016.
Crossref Google Scholar
[13]

S. Al-Agtash, "Electricity agents in smart grid markets," Computers in Industry, vol. 64, no. 3, pp. 235–241, Apr. 2013.
Crossref Google Scholar
[14]

M. L. Zheng, C. J. Meinrenken, and K. S. Lackner, "Agent-based model for electricity consumption and storage to evaluate economic viability of tariff arbitrage for residential sector demand response," Applied Energy, vol. 126, pp. 297–306, Aug. 2014.
Crossref Google Scholar
[15]

M. Mahvi and M. M. Ardehali, "Optimal bidding strategy in a competitive electricity market based on agent-based approach and numerical sensitivity analysis," Energy, vol. 36, no. 11, pp. 6367–6374, Nov. 2011.
Crossref Google Scholar
[16]

F. Silva, B. Teixeira, T. Pinto, G. Santos, Z. Vale, and I. Praça, "Generation of realistic scenarios for multi-agent simulation of electricity markets," Energy, vol. 116, pp. 128–139, Dec. 2016.
Crossref Google Scholar
[17]
L. Buşoniu, R. Babuška, and B. De Schutter, "Multi-agent reinforcement learning: an overview, " in Innovations in Multi-Agent Systems and Application-1, D. Srinivasan and L. C. Jain, Eds. Berlin, Heidelberg: Springer-Verlag, 2010, pp. 183–221.
Crossref
[18]

G. Weiss, Multiagent Systems: A Modern Approach to Distributed Artificial Intelligence. The MIT Press, USA:MA, 1999.
[19]

R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., Cambridge: The MIT Press, 2018.
[20]
M. L. Littman, "Markov games as a framework for multi-agent reinforcement learning, " in Machine Learning Proceedings 1994: Proceedings of the Eleventh International Conference, Rutgers University, New Brunswick, NJ, July 10–13, 1994, W. W. Cohen and H. Hirsh, Eds. Amsterdam: Elsevier, 1994, pp. 157–163.
Crossref
[21]

X. L. Wang and H. S. Sun, "Completely model-free RL-based consensus of continuous-time multi-agent systems," Applied Mathematics and Computation, vol. 382, pp. 125312, Oct. 2020.
Crossref Google Scholar
[22]

D. Wang, B. Liu, H. J. Jia, Z. Y. Zhang, J. C. Chen, D. Y. Huang, "Peer-to-peer electricity transaction decision of user-side smart energy system based on SARSA reinforcement learning method," CSEE Journal of Power and Energy Systems, pp. 1–11, Nov. 2020.
Google Scholar
[23]

Y. Zhang, Z. F. Zhang, Q. Y. Yang, D. An, D. H. Li, and C. Li, "EV charging bidding by multi-DQN reinforcement learning in electricity auction market," Neurocomputing, vol. 397, pp. 404–414, Jul. 2020.
Crossref Google Scholar
[24]
J. L. Hu and M. P. Wellman, "Multiagent reinforcement learning: theoretical framework and an algorithm, " in Proceedings of the 15th International Conference on Machine Learning (ICML 1998), Madison, Wisconsin, USA, Jul. 24–27, 1998.
[25]

T. Ding, Z. Y. Zeng, J. W. Bai, B. Y. Qin, Y. H. Yang, and M. Shahidehpour, "Optimal electric vehicle charging strategy with markov decision process and reinforcement learning technique," IEEE Transactions on Industry Applications, vol. 56, no. 5, pp. 5811–5823, Sep./Oct. 2020.
Crossref Google Scholar
[26]

H. X. Zang, L. L. Cheng, T. Ding, K. W. Cheung, M. M. Wang, Z. N. Wei, and G. Q. Sun "Application of functional deep belief network for estimating daily global solar radiation: A case study in China," Energy, vol. 191, pp. 116502, Jan. 2020.
Crossref Google Scholar
[27]

N. P. Yu, C. C. Liu, and J. Price, "Evaluation of market rules using a multi-agent system method," IEEE Transactions on Power Systems, vol. 25, no. 1, pp. 470–479, Feb. 2010.
Crossref Google Scholar
[28]

K. Dehghanpour, M. H. Nehrir, J. W. Sheppard, and N. C. Kelly, "Agent-based modeling in electrical energy markets using dynamic Bayesian networks," IEEE Transactions on Power Systems, vol. 31, no. 6, pp. 4744–4754, Nov. 2016.
Crossref Google Scholar
[29]

V. Nanduri and T. K. Das, "A reinforcement learning model to assess market power under auction-based energy pricing," IEEE Transactions on Power Systems, vol. 22, no. 1, pp. 85–95, Feb. 2007.
Crossref Google Scholar
[30]

A. Banal-Estañol and A. R. Micola, "Behavioural simulations in spot electricity markets," European Journal of Operational Research, vol. 214, no. 1, pp. 147–159, Oct. 2011.
Crossref Google Scholar
[31]

X. H. Fang, J. K. Wang, G. R. Song, Y. H. Han, and Q. A. Zhao, and Z. Cao, "Multi-agent reinforcement learning approach for residential microgrid energy scheduling," Energies, vol. 13, no. 1, pp. 123, Jan. 2020.
Crossref Google Scholar
[32]

H. Kazmi, J. Suykens, A. Balint, and J. Driesen, "Multi-agent reinforcement learning for modeling and control of thermostatically controlled loads," Applied Energy, vol. 238, pp. 1022–1035, Mar. 2019.
Crossref Google Scholar
[33]
M. Peters, Machine learning algorithms for smart electricity markets. Erasmus Research Institute of Management, NL: Rotterdam, No. EPS-2014–332-LIS, Feb. 2015.
[34]

C. Camerer and T. H. Ho, "Experience-weighted attraction learning in normal-form games," Econometrica, vol. 67, no. 4, pp. 827–874, Jul. 1999.
Crossref Google Scholar
[35]

C. Camerer and T. H. Ho, "Experience-weighted attraction learning in coordination games: probability rules, heterogeneity, and time-variation," Journal of Mathematical Psychology, vol. 42, no. 2–3, pp. 305–326, Jun. 1998.
Crossref Google Scholar
[36]

C. Camerer and T. H. Ho, "Experience-weighted attraction learning in normal form games," Econometrica, vol. 67, no. 4, pp. 827–874, Jul. 1999.
Crossref Google Scholar
[37]

W. van der Horst, M. A. L. M. van Assen, and C. C. P. Snijders, "Analyzing behavior implied by EWA learning: An emphasis on distinguishing reinforcement from belief learning," Journal of mathematical Psychology, vol. 54, no. 2, pp. 222–229, Apr. 2010.
Crossref Google Scholar
[38]

T. H. Ho, C. F. Camerer, and J. K. Chong, "Self-tuning experience weighted attraction learning in games," Journal of Economic Theory, vol. 133, no. 1, pp. 177–198, Mar. 2007.
Crossref Google Scholar
[39]

M. Bigoni and M. Fort, "Information and learning in oligopoly: an experiment," Games and Economic Behavior, vol. 81, pp. 192–214, Sep. 2013.
Crossref Google Scholar
[40]

C. F. Camerer, T. H. Ho, and J. K. Chong, "Sophisticated experience-weighted attraction learning and strategic teaching in repeated games," Journal of Economic Theory, vol. 104, no. 1, pp. 137–188, May. 2002.
Crossref Google Scholar
[41]

J. K. Chong, C. F. Camerer, and T. H. Ho, "A learning-based model of repeated games with incomplete information," Games and Economic Behavior, vol. 55, no. 2, pp. 340–371, May. 2006.
Crossref Google Scholar
[42]

Q. Y. Li, H. P. Yao, T. L. Mai, C. X. Jiang, and Y. Zhang, "Reinforcement-learning and belief-learning-based double auction mechanism for edge computing resource allocation," IEEE Internet of Things Journal, vol. 7, no. 7, pp. 5976–5985, Jul. 2020.
Crossref Google Scholar
[43]

C. M. Anderson and C. F. Camerer, "Experience-weighted attraction learning in sender-receiver signaling games," Economic Theory, vol. 16, no. 3. pp. 689–718, Nov. 2000.
Crossref Google Scholar
[44]

Y. Sun and J. S. Qian, "EWA Selection strategy with channel handoff scheme in cognitive radio," Wireless Personal Communications, vol. 87, no. 1, pp. 17–28, Mar. 2016.
Crossref Google Scholar
[45]

M. Uwasu, "Estimating the experience-weighted attractions for the migration-emission game," Theoretical Economics Letters, vol. 2, no. 5, pp. 494–501, Dec. 2012.
Crossref Google Scholar
[46]

M. S. Iftekhar and A. Hailu, "Experience Weighted Attraction (EWA) learning parameter value choice for multiple-unit iterative combinatorial auction games," Journal of Computational Optimization in Economics and Finance, vol. 3, no. 1, pp. 1–11, Jan. 2012.
Google Scholar
[47]

Z. H. Zhang, S. Chandra, A. Kayser, M. Hsu, and J. L. Warren, "A hierarchical Bayesian implementation of the experience- weighted attraction model," Computational Psychiatry, vol. 4, pp. 40–60, Nov. 2020.
Crossref Google Scholar
[48]

Z. X. Jing and Y. Yang, "Application of the EWA Algorithm in electricity market simulation," Automation of Electric Power Systems, vol. 34, no. 24, pp. 46–50, Dec. 2010.
Google Scholar
[49]

C. J. C. H. Watkins and P. Dayan, "Technical note: Q-learning," Machine Learning, vol. 8, no. 3–4, pp. 279–292, May. 1992.
Crossref Google Scholar
[50]
G. F. Xiong, T. Hashiyama, and S. Okuma, "An electricity supplier bidding strategy through Q-learning, " in Proceedings of 2002 IEEE Power Engineering Society Summer Meeting, Chicago, IL, USA, Jul. 21–25, 2002.
[51]

J. D. Wang, J. H. Wu, and Y. B. Che, "Agent and system dynamics-based hybrid modeling and simulation for multilateral bidding in electricity market," Energy, vol. 180, pp. 444–456, Aug. 2019.
Crossref Google Scholar
[52]
A. C. Tellidou and A. G. Bakirtzis, "Multi-agent reinforcement learning for strategic bidding in power markets, " in Proceedings of the 3rd International IEEE Conference Intelligent Systems, London, UK, Sep. 4–6, 2006.
Crossref
[53]
N. Keseric, M. Saguan, and G. Hadjee, "Computer as thinking agent: Modeling the European power markets using agent-based simulation, " in Proceedings of 2005 International Conference on "Computer as a Tool", Belgrade, Serbia, Nov. 22–24, 2005.
Crossref
CSEE Journal of Power and Energy Systems
Volume 9 Issue 3,
May 2023
Pages 1051-1065
DOI: 10.17775/CSEEJPES.2020.02820
Cite this article:
Wang J, Wu J, Kong X. Multi-agent Simulation for Strategic Bidding in Electricity Markets Using Reinforcement Learning. CSEE Journal of Power and Energy Systems, 2023, 9(3): 1051-1065. https://doi.org/10.17775/CSEEJPES.2020.02820

298

Views

8

Downloads

4

Crossref

5

Web of Science

10

Scopus

0

CSCD

Altmetrics
Received: 26 June 2020
Revised: 07 February 2021
Accepted: 12 May 2021
Published: 13 November 2021
© 2020 CSEE.

This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Return