Pyramid Image Fusion Based on Contourlet and Enhanced Structural Decomposition

Document Type : Power Article

Authors

1 Babol Noshirvani University of Technology, Babol

2 Babol Noshirvani University of Technology, Faculty of Electrical and Computer Eng.

Abstract

Recently, a method for multi-exposure images fusion based on structural decomposition of images into three parts including signal strength, signal structure and signal mean has been introduced. In this paper, we seek to use this decomposition, for images fusion in other fields, including multimodal medical, multi-focus, and infrared and visible images. To increase the fusion quality, besides the introduction of the proposed weighting factor in the structural decomposition, contourlet transformation and the pyramidal structure have also been used. First, each of the K input images are represented into low frequency and high frequency subbands, by using contourlet transform. Then, all the corresponding subbands (resulting from the same scales and directions) are fused with each other, separately and in an iterative process. In this iterative process, first, a separate pyramid structure (including approximation and detail layers) is created for each of the corresponding K subbands. These layers are obtained by the down-sampling of subbands and structural separation based on the proposed new weighting factor. Then, the fusion is performed in the reverse direction of the pyramidal structure and the fused image of the K corresponding subband is obtained. By repeating this process, the fused image will be obtained for all the corresponding subbands. At the end, the final fused image is obtained by the inverse contourlet transformation on the fused images of the subbands. Several visual and quantitative comparisons, with 7 common methods in this field, have been made. In the visual aspect, the proposed method shows the highest quality.

Keywords

Main Subjects

References

[1] Kaur, H., Koundal, D., & Kadyan, V., "Image fusion techniques: a survey." Archives of computational methods in Engineering 28, no. 7, 2021, pp. 4425-4447.
[2] Azam, M. A., Khan, K. B., Salahuddin, S., Rehman, E., Khan, S. A., Khan, M. A. & Gandomi, A. H., "A review on multimodal medical image fusion: Compendious analysis of medical modalities, multimodal databases, fusion techniques and quality metrics." Computers in Biology and Medicine 144, 2022, pp. 105253.
[3] Song L, Lin Y, Feng W, Zhao M , "A novel automatic weighted image fusion algorithm." In 2009 International Workshop on Intelligent Systems and Applications, IEEE, 2009, pp. 1-4.
[4] Singh, H., Kumar, V., & Bhooshan, S., "Weighted least squares based detail enhanced exposure fusion." International Scholarly Research Notices, 2014.
[5] Bavirisetti, D. P., Xiao, G., & Liu, G., "Multi-sensor image fusion based on fourth order partial differential equations." In 2017 20th International conference on information fusion (Fusion), IEEE, 2017, pp. 1-9.
[6] Zhang, X., Lin, H., Kang, X., & Li, S., "Multi-modal image fusion with KNN matting." In Chinese Conference on Pattern Recognition, Springer, Berlin, Heidelberg, 2014. pp. 89-96.
[7] Shreyamsha Kumar, B. K., "Image fusion based on pixel significance using cross bilateral filter." Signal, image and video processing 9, no. 5, 2015, pp. 1193-1204.
[8] Zhan, K., Xie, Y., Wang, H., & Min, Y., "Fast filtering image fusion." Journal of Electronic Imaging 26, no. 6, 2017, pp. 063004.
[9] Ma, J., & Zhou, Y., "Infrared and visible image fusion via gradientlet filter." Computer Vision and Image Understanding 197, 2020, pp. 103016.
[10] Zhang, Y., Wang, M., Xia, X., Sun, D., Zhou, X., Wang, Y., & Huang, G., "Medical image fusion based on quasi-cross bilateral filtering." Biomedical Signal Processing and Control, 80, 2023, pp. 104259.
[11] Zhang, X., Dai, X., Zhang, X., & Jin, G., "Joint Principal Component Analysis and Total Variation for Infrared and Visible Image Fusion." Infrared Physics & Technology, 2022, pp. 104523.
[12] Yang, Y., Cao, S., Wan, W., & Huang, S., "Multi-modal medical image super-resolution fusion based on detail enhancement and weighted local energy deviation." Biomedical Signal Processing and Control, 80, 2023, pp. 104387.
[13] Li, S., Kang, X., & Hu, J., "Image fusion with guided filtering." IEEE Transactions on Image processing 22, no. 7, 2013, pp. 2864-2875.
[14] Naidu, V. P. S., "Image fusion technique using multi-resolution singular value decomposition." Defence Science Journal 61, no. 5, 2011, pp. 479.
[15] Zhan, K., Li, Q., Teng, J., Wang, M., & Shi, J., "Multifocus image fusion using phase congruency." Journal of Electronic Imaging 24, no. 3, 2015, pp. 033014.
[16] Hayat, N., & Imran, M., "Ghost-free multi exposure image fusion technique using dense SIFT descriptor and guided filter." Journal of Visual Communication and Image Representation 62, 2019, pp. 295-308.
[17] Zhang, Q., & Guo, B. L., "Multifocus image fusion using the nonsubsampled contourlet transform." Signal processing 89, no. 7, 2009, pp. 1334-1346.
[18] Li, X., Wan, W., Zhou, F., Cheng, X., Jie, Y., & Tan, H., "Medical image fusion based on sparse representation and neighbor energy activity." Biomedical Signal Processing and Control, 80, 2023, pp. 104353.
[19] Diwakar, M., Singh, P., & Shankar, A., "Multi-modal medical image fusion framework using co-occurrence filter and local extrema in NSST domain." Biomedical Signal Processing and Control, 68, 2021, PP. 102788.
[20] Nencini, F., Garzelli, A., Baronti, S., & Alparone, L., "Remote sensing image fusion using the curvelet transform." Information fusion 8, no. 2, 2007, pp. 143-156.
[21] Lewis, J. J., O’Callaghan, R. J., Nikolov, S. G., Bull, D. R., & Canagarajah, N., "Pixel-and region-based image fusion with complex wavelets." Information fusion 8, no. 2, 2007, pp. 119-130.
[22] Shreyamsha Kumar, B. K., "Multifocus and multispectral image fusion based on pixel significance using discrete cosine harmonic wavelet transform." Signal, Image and Video Processing 7, no. 6, 2013, pp. 1125-1143.
[23] Bavirisetti, D. P., Xiao, G., Zhao, J., Dhuli, R., & Liu, G., "Multi-scale guided image and video fusion: A fast and efficient approach." Circuits, Systems, and Signal Processing 38, no. 12, 2019, pp. 5576-5605.
[24] Li, H., Ma, K., Yong, H., & Zhang, L., "Fast multi-scale structural patch decomposition for multi-exposure image fusion." IEEE Transactions on Image Processing 29, 2020, pp. 5805-5816.
[25] Ma, K., Li, H., Yong, H., Wang, Z., Meng, D., & Zhang, L., "Robust multi-exposure image fusion: a structural patch decomposition approach." IEEE Transactions on Image Processing, 26, no. 5, 2017, pp. 2519-2532.
[26] Zhang, H., Xu, H., Tian, X., Jiang, J., & Ma, J., "Image fusion meets deep learning: A survey and perspective." Information Fusion 76, 2021, pp. 323-336.
[27] مرضیه زارع نظری، محسن سرداری زارچی، سیما عمادی و هادی پورمحمدی، "چارچوب برای استخراج آناتومی و طبقه بندی تصاویر پشه با رویکرد یادگیری عمیق"، نشریه مدل‌سازی در مهندسی، دوره 20، شماره 70، بهار 1401.
[28] مسلم سردشتی بیرجندی، حسین رحمانی و سعید فراهت، "کاربرد شبکه­های عصبی عمیق در طبقه بندی تصاویر آسیب­های شبکه فاضلاب و مشخص کردن مسیرهای بحرانی آنها"، نشریه مدل‌سازی در مهندسی، دوره 20، شماره 70، بهار 1401.
[29] Amin-Naji, M., Aghagolzadeh, A., & Ezoji, M., "Ensemble of CNN for multi-focus image fusion." Information fusion 51, 2019, pp. 201-214.
[30] Gao, Y., Ma, S., Liu, J., & Xiu, X., "Fusion-UDCGAN: Multifocus Image Fusion via a U-Type Densely Connected Generation Adversarial Network." IEEE Transactions on Instrumentation and Measurement, 71, 2022, pp. 1-13.
[31] Xia, K. J., Yin, H. S., & Wang, J. Q., "A novel improved deep convolutional neural network model for medical image fusion." Cluster Computing 22, no. 1, 2019, pp. 1515-1527.
[32] Zhang, B., Jiang, C., Hu, Y., & Chen, Z., "Medical image fusion based a densely connected convolutional networks." In 2021 IEEE 5th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), IEEE, 2021, vol. 5, pp. 2164-2170
[33] Zhang, H., Yuan, J., Tian, X., & Ma, J., "GAN-FM: Infrared and visible image fusion using GAN with full-scale skip connection and dual Markovian discriminators." IEEE Transactions on Computational Imaging 7, 2021, pp. 1134-1147.
[34] Upla, K. P., Joshi, M. V., & Gajjar, P. P., "An edge preserving multiresolution fusion: Use of contourlet transform and MRF prior." IEEE Transactions on Geoscience and Remote Sensing 53, no. 6, 2014, pp. 3210-3220.
[35] Zhang, Q., & Guo, B. L., "Multifocus image fusion using the nonsubsampled contourlet transform." Signal processing 89, no. 7, 2009, pp. 1334-1346.
[36] Da Cunha, A. L., Zhou, J., & Do, M. N., "The nonsubsampled contourlet transform: theory, design, and applications." IEEE transactions on image processing 15, no. 10, 2006, pp. 3089-3101.
[37] He, K., Sun, J., & Tang, X., "Guided image filtering." IEEE transactions on pattern analysis and machine intelligence 35, no. 6, 2012, pp. 1397-1409.
[38] Yin, M., Liu, X., Liu, Y., & Chen, X., "Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain." IEEE Transactions on Instrumentation and Measurement 68, no. 1, 2018, pp. 49-64.
[39] Xydeas, C. S., & Petrovic, V., "Objective image fusion performance measure." Electronics letters 36, no. 4, 2000, pp. 308-309.
[40] Zheng, Y., Essock, E. A., Hansen, B. C., & Haun, A. M., "A new metric based on extended spatial frequency and its application to DWT based fusion algorithms." Information Fusion 8, no. 2, 2007, pp. 177-192.
[41] Zhao, J., Laganiere, R., & Liu, Z., "Performance assessment of combinative pixel-level image fusion based on an absolute feature measurement." Int. J. Innov. Comput. Inf. Control 3, no. 6, 2007, pp. 1433-1447.
[42] Piella, G., & Heijmans, H., "A new quality metric for image fusion." In Proceedings 2003 international conference on image processing IEEE, (Cat. No. 03CH37429), 2003, vol. 3, pp. III-173.
[43] Chen, H., & Varshney, P. K., "A human perception inspired quality metric for image fusion based on regional information." Information fusion 8, no. 2, 2007, pp. 193-207.
[44] Li, S., Kwok, J. T., & Wang, Y., "Combination of images with diverse focuses using the spatial frequency." Information fusion 2, no. 3, 2001, pp. 169-176.
[45] Neurological, M. B. I. C. M. (2020). Institute, McGill University, Montreal, QC, Canada. BrainWeb.. https:// brain web. bic. mni. mcgill. ca/ brain web/. Accessed April. 1, 2022.
[46] Johnson, K. A., & Becker, J. A. (1999). The Whole Brain Atlas, Harvard Medical School. https:// www. med. harva rd. edu/ aanlib. April. 1, 2022.
[47] Liu, Y., Wang, L., Cheng, J., Li, C., & Chen, X., "Multi-focus image fusion: A survey of the state of the art." Information Fusion 64, 2020, pp. 71-91.
[48] Alexander Toet et al. Tno image fusion dataset. https://doi.org/10.6084/m9.figshare.1008029.v1 , April. 1, 2022.
[49] اصغر زارع و علی محمدزاده، "حذف نویز ضربه‌ای از تصاویر دیجیتالی مبتنی بر تخمین توزیع مکانی نویزها"، نشریه مدل‌سازی در مهندسی، دوره 12، شماره 39، زمستان 1393، صفحه 13- 29.
Journal of Modeling in Engineering
Volume 21, Issue 73
June 2023
Pages 71-92

Files

Share

How to cite

Statistics