The development of hand gesture recognition systems has gained more attention in recent days, due to its support of modern human-computer interfaces. Moreover, sign language recognition is mainly developed for enabling communication between deaf and dumb people. In conventional works, various image processing techniques like segmentation, optimization, and classification are deployed for hand gesture recognition. Still, it limits the major problems of inefficient handling of large dimensional datasets and requires more time consumption, increased false positives, error rate, and misclassification outputs. Hence, this research work intends to develop an efficient hand gesture image recognition system by using advanced image processing techniques. During image segmentation, skin color detection and morphological operations are performed for accurately segmenting the hand gesture portion. Then, the Heuristic Manta-ray Foraging Optimization (HMFO) technique is employed for optimally selecting the features by computing the best fitness value. Moreover, the reduced dimensionality of features helps to increase the accuracy of classification with a reduced error rate. Finally, an Adaptive Extreme Learning Machine (AELM) based classification technique is employed for predicting the recognition output. During results validation, various evaluation measures have been used to compare the proposed model’s performance with other classification approaches.