FCDFusion: A fast, low color deviation method for fusing visible and infrared image pairs
)Graphical Abstract
Abstract
Visible and infrared image fusion (VIF) aims to combine information from visible and infrared images into a single fused image. Previous VIF methods usually employ a color space transformation to keep the hue and saturation from the original visible image. However, for fast VIF methods, this operation accounts for the majority of the calculation and is the bottleneck preventing faster processing. In this paper, we propose a fast fusion method, FCDFusion, with little color deviation. It preserves color information without color space transformations, by directly operating in RGB color space. It incorporates gamma correction at little extra cost, allowing color and contrast to be rapidly improved. We regard the fusion process as a scaling operation on 3D color vectors, greatly simplifying the calculations. A theoretical analysis and experiments show that our method can achieve satisfactory results in only 7 FLOPs per pixel. Compared to state-of-the-art fast, color-preserving methods using HSV color space, our method provides higher contrast at only half of the computational cost. We further propose a new metric, color deviation, to measure the ability of a VIF method to preserve color. It is specifically designed for VIF tasks with color visible-light images, and overcomes deficiencies of existing VIF metrics used for this purpose. Our code is available at https://github.com/HeasonLee/FCDFusion.
Keywords
References
Ma, J.; Ma, Y.; Li, C. Infrared and visible image fusion methods and applications: A survey. Information Fusion Vol. 45, 153–178, 2019.
Cao, Y.; Guan, D.; Huang, W.; Yang, J.; Cao, Y.; Qiao, Y. Pedestrian detection with unsupervised multispectral feature learning using deep neural networks. Information Fusion Vol. 46, 206–217, 2019.
Gao, S.; Cheng, Y.; Zhao, Y. Method of visual and infrared fusion for moving object detection. Optics Letters Vol. 38, No. 11, Article No. 1981, 2013.
Han, J.; Bhanu, B. Fusion of color and infrared video for moving human detection. Pattern Recognition Vol. 40, No. 6, 1771–1784, 2007.
Ulusoy, I.; Yuruk, H. New method for the fusion of complementary information from infrared and visual images for object detection. IET Image Processing Vol. 5, No. 1, 36–48, 2011.
Liu, H.; Sun, F. Fusion tracking in color and infrared images using joint sparse representation. Science China Information Sciences Vol. 55, No. 3, 590–599, 2012.
Smith, D.; Singh, S. Approaches to multisensor data fusion in target tracking: A survey. IEEE Transactions on Knowledge and Data Engineering Vol. 18, No. 12, 1696–1710, 2006.
Kong, S. G.; Heo, J.; Abidi, B. R.; Paik, J.; Abidi, M. A. Recent advances in visual and infrared face recognition—A review. Computer Vision and Image Understanding Vol. 97, No. 1, 103–135, 2005.
Simone, G.; Farina, A.; Morabito, F. C.; Serpico, S. B.; Bruzzone, L. Image fusion techniques for remote sensing applications. Information Fusion Vol. 3, No. 1, 3–15, 2002.
Yin, S.; Cao, L.; Ling, Y.; Jin, G. One color contrast enhanced infrared and visible image fusion method. Infrared Physics & Technology Vol. 53, No. 2, 146–150, 2010.
Fu, M. Y.; Zhao, C. Fusion of infrared and visible images based on the second generation curvelet transform. Journal of Infrared and Millimeter Waves Vol. 28, No. 4, 254–258, 2009. (in Chinese)
Li, H.; Ding, W.; Cao, X.; Liu, C. Image registration and fusion of visible and infrared integrated camera for medium-altitude unmanned aerial vehicle remote sensing. Remote Sensing Vol. 9, No. 5, Article No. 441, 2017.
Miao, Q. G.; Wang, B. S. Multi-sensor image fusion based on improved Laplacian pyramid transform. Acta Optica Sinica Vol. 27, No. 9, 1605–1610, 2007. (in Chinese)
Pajares, G.; Manuel de la Cruz, J. A wavelet-based image fusion tutorial. Pattern Recognition Vol. 37, No. 9, 1855–1872, 2004.
Naidu, V. P. S. Image fusion technique using multi-resolution singular value decomposition. Defence Science Journal Vol. 61, No. 5, Article No. 479, 2011.
Li, H.; Wu, X. J.; Durrani, T. S. Infrared and visible image fusion with ResNet and zero-phase component analysis. Infrared Physics & Technology Vol. 102, Article No. 103039, 2019.
Liu, Y.; Chen, X.; Cheng, J.; Peng, H.; Wang, Z. Infrared and visible image fusion with convolutional neural networks. International Journal of Wavelets, Multiresolution and Information Processing Vol. 16, No. 3, Article No. 1850018, 2018.
Xu, H.; Gong, M.; Tian, X.; Huang, J.; Ma, J. CUFD: An encoder–decoder network for visible and infrared image fusion based on common and unique feature decomposition. Computer Vision and Image Understanding Vol. 218, Article No. 103407, 2022.
Tang, L.; Yuan, J.; Zhang, H.; Jiang, X.; Ma, J. PIAFusion: A progressive infrared and visible image fusion network based on illumination aware. Information Fusion Vol. 83, 79–92, 2022.
Tang, L.; Yuan, J.; Ma, J. Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network. Information Fusion Vol. 82, 28–42, 2022.
Wang, Z.; Gong, C. A multi-faceted adaptive image fusion algorithm using a multi-wavelet-based matching measure in the PCNN domain. Applied Soft Computing Vol. 61, 1113–1124, 2017.
Yin, M.; Duan, P.; Liu, W.; Liang, X. A novel infrared and visible image fusion algorithm based on shift-invariant dual-tree complex shearlet transform and sparse representation. Neurocomputing Vol. 226, 182–191, 2017.
Liu, Y.; Liu, S.; Wang, Z. A general framework for image fusion based on multi-scale transform and sparse representation. Information Fusion Vol. 24, 147–164, 2015.
Zhang, X.; Ma, Y.; Fan, F.; Zhang, Y.; Huang, J. Infrared and visible image fusion via saliency analysis and local edge-preserving multi-scale decomposition. Journal of the Optical Society of America A Vol. 34, No. 8, Article No. 1400, 2017.
Luo, Y.; He, K.; Xu, D.; Yin, W.; Liu, W. Infrared and visible image fusion based on visibility enhancement and hybrid multiscale decomposition. Optik Vol. 258, Article No. 168914, 2022.
Yin, W.; He, K.; Xu, D.; Luo, Y.; Gong, J. Significant target analysis and detail preserving based infrared and visible image fusion. Infrared Physics & Technology Vol. 121, Article No. 104041, 2022.
Van Aardt, J. Assessment of image fusion procedures using entropy, image quality, and multispectral classification. Journal of Applied Remote Sensing Vol. 2, No. 1, Article No. 023522, 2008.
Cui, G.; Feng, H.; Xu, Z.; Li, Q.; Chen, Y. Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition. Optics Communications Vol. 341, 199–209, 2015.
Wang, Z.; Bovik, A. C.; Sheikh, H. R.; Simoncelli, E. P. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing Vol. 13, No. 4, 600–612, 2004.
Jagalingam, P.; Hegde, A. V. A review of quality metrics for fused image. Aquatic Procedia Vol. 4, 133–142, 2015.
Mu, Q.; Wang, X.; Wei, Y.; Li, Z. Low and non-uniform illumination color image enhancement using weighted guided image filtering. Computational Visual Media Vol. 7, No. 4, 529–546, 2021.
Schwarz, M. W.; Cowan, W. B.; Beatty, J. C. An experimental comparison of RGB, YIQ, LAB, HSV, and opponent color models. ACM Transactions on Graphics Vol. 6, No. 2, 123–158, 1987.
Smith, A. R. Color gamut transform pairs. ACM SIGGRAPH Computer Graphics Vol. 12, No. 3, 12–19, 1978.
Liu, Z.; Blasch, E.; Xue, Z.; Zhao, J.; Laganiere, R.; Wu, W. Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 34, No. 1, 94–109, 2012.
Bulanon, D. M.; Burks, T. F.; Alchanatis, V. Image fusion of visible and thermal images for fruit detection. Biosystems Engineering Vol. 103, No. 1, 12–22, 2009.
Qu, G.; Zhang, D.; Yan, P. Information measure for performance of image fusion. Electronics Letters Vol. 38, No. 7, 313–315, 2002.
Rajalingam, B.; Priya, R. Hybrid multimodality medical image fusion technique for feature enhancement in medical diagnosis. International Journal of Engineering Science Invention Vol. 2, 52–60, 2018.
Rao, Y. J. In-fibre Bragg grating sensors. Measurement Science and Technology Vol. 8, No. 4, 355–375, 1997.
Eskicioglu, A. M.; Fisher, P. S. Image quality measures and their performance. IEEE Transactions on Communications Vol. 43, No. 12, 2959–2965, 1995.
Xydeas, C. S.; Petrović, V. Objective image fusion performance measure. Electronics Letters Vol. 36, No. 4, 308–309, 2000.
Chen, Y.; Blum, R. S. A new automated quality assessment algorithm for image fusion. Image and Vision Computing Vol. 27, No. 10, 1421–1432, 2009.
Chen, H.; Varshney, P. K. A human perception inspired quality metric for image fusion based on regional information. Information Fusion Vol. 8, No. 2, 193–207, 2007.
Poynton, C. Digital Video and HD: Algorithms and Interfaces. Morgan Kaufmann Publishers Inc., 2012.
京公网安备11010802044758号