Federated Learning (FL) allows the Internet of bioMedical Things (IoMT) devices to collaboratively train a global model without centralizing data, yet the heterogeneity of biomedical images poses challenges in achieving optimal performance for each IoMT device. Although many personalized FL methods have been proposed, their adaptability is often limited to data with certain levels of heterogeneity. In this paper, we propose a novel FL method to handle heterogeneous images in IoMT. First, we propose a gradient-fusion method that integrates both shared and local gradients into the locally adapted model for improved personalization. The shared gradients, aggregated from all IoMT devices through a learnable matrix, capture collective intelligence, while the local gradients, originating from each device’s data, reflect individual data distributions. This dynamic integration of collective and device-specific insights effectively mitigates data heterogeneity. Second, we propose a privacy-enhanced approach that delegates part of the gradient computation to devices, thereby protecting data privacy without compromising the efficacy of the gradient-fusion process. Finally, for enhanced performance, we introduce a layer-wise aggregation method to precisely measure the contribution of different layers in local models. Extensive evaluations on imaging datasets, featuring various types and degrees of data heterogeneity, demonstrate the superior performance of our methods over existing baselines.