Particle swarm optimization (PSO) algorithms have been successfully used for various complex optimization problems. However, balancing the diversity and convergence is still a problem that requires continuous research. Therefore, an evolutionary experience-driven particle swarm optimization with dynamic searching (EEDSPSO) is proposed in this paper. For purpose of extracting the effective information during population evolution, an adaptive framework of evolutionary experience is presented. And based on this framework, an experience-based neighborhood topology adjustment (ENT) is used to control the size of the neighborhood range, thereby effectively keeping the diversity of population. Meanwhile, experience-based elite archive mechanism (EEA) adjusts the weights of elite particles in the late evolutionary stage, thus enhancing the convergence of the algorithm. In addition, a Gaussian crisscross learning strategy (GCL) adopts cross-learning method to further balance the diversity and convergence. Finally, extensive experiments use the CEC2013 and CEC2017. The experiment results show that EEDSPSO outperforms current excellent PSO variants.