In this paper, a theoretical framework of Multi-agent Simulation (MAS) is proposed for strategic bidding in electricity markets using reinforcement learning, which consists of two parts: one is a MAS system used to simulate the competitive bidding of the actual electricity market; the other is an adaptive learning strategy bidding system used to provide agents with more intelligent bidding strategies. An Experience-Weighted Attraction (EWA) reinforcement learning algorithm (RLA) is applied to the MAS model and a new MAS method is presented for strategic bidding in electricity markets using a new Improved EWA (IEWA). From both qualitative and quantitative perspectives, it is compared with three other MAS methods using the Roth-Erev (RE), Q-learning and EWA. The results show that the performance of the MAS method using IEWA is proved to be better than the others. The four MAS models using four RLAs are built for strategic bidding in electricity markets. Through running the four MAS models, the rationality and correctness of the four MAS methods are verified for strategic bidding in electricity markets using reinforcement learning.