Telemarketing is a well-established marketing approach to offering products and services to prospective customers. The effectiveness of such an approach, however, is highly dependent on the selection of the appropriate consumer base, as reaching uninterested customers will induce annoyance and consume costly enterprise resources in vain while missing interested ones. The introduction of business intelligence and machine learning models can positively influence the decision-making process by predicting the potential customer base, and the existing literature in this direction shows promising results. However, the selection of influential features and the construction of effective learning models for improved performance remain a challenge. Furthermore, from the modelling perspective, the class imbalance nature of the training data, where samples with unsuccessful outcomes highly outnumber successful ones, further compounds the problem by creating biased and inaccurate models. Additionally, customer preferences are likely to change over time due to various reasons, and/or a fresh group of customers may be targeted for a new product or service, necessitating model retraining which is not addressed at all in existing works. A major challenge in model retraining is maintaining a balance between stability (retaining older knowledge) and plasticity (being receptive to new information). To address the above issues, this paper proposes an ensemble machine learning model with feature selection and oversampling techniques to identify potential customers more accurately. A novel online learning method is proposed for model retraining when new samples are available over time. This newly introduced method equips the proposed approach to deal with dynamic data, leading to improved readiness of the proposed model for practical adoption, and is a highly useful addition to the literature. Extensive experiments with real-world data show that the proposed approach achieves excellent results in all cases (e.g., 98.6% accuracy in classifying customers) and outperforms recent competing models in the literature by a considerable margin of 3% on a widely used dataset.