American Institute of Mathematical Sciences

Advanced
December  2021, 15(6): 1409-1419. doi: 10.3934/ipi.2021067

Image fusion network for dual-modal restoration

Ying Zhang 1, , Xuhua Ren 2, , Bryan Alexander Clifford 3, , Qian Wang 2, and  Xiaoqun Zhang 4,,

1. 

School of Statistics and Mathematics, Shanghai Lixin University of Accounting and Finance, Shanghai, 201209, China
2. 

School of Biomedical Engineering, Shanghai Jiao Tong University, China
3. 

Department of Electrical and Computer Engineering, Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign, USA
4. 

School of Mathematical Sciences, MOE-LSC, Shanghai Jiao Tong University, China, Institute of Natural Sciences, Shanghai Jiao Tong University, China

*Corresponding author: Xiaoqun Zhang

Received  November 2021 Revised  August 2021 Published  December 2021 Early access  October 2021

In recent years multi-modal data processing methods have gained considerable research interest as technological advancements in imaging, computing, and data storage have made the collection of redundant, multi-modal data more commonplace. In this work we present an image restoration method tailored for scenarios where pre-existing, high-quality images from different modalities or contrasts are available in addition to the target image. Our method is based on a novel network architecture which combines the benefits of traditional multi-scale signal representation, such as wavelets, with more recent concepts from data fusion methods. Results from numerical simulations in which T1-weighted MRI images are used to restore noisy and undersampled T2-weighted images demonstrate that the proposed network successfully utilizes information from high-quality reference images to improve the restoration quality of the target image beyond that of existing popular methods.

Keywords: Image restoration network, dual-modal imaging, multi-scale image representation, information fusion.
Mathematics Subject Classification: Primary: 68U10.
Citation: Ying Zhang, Xuhua Ren, Bryan Alexander Clifford, Qian Wang, Xiaoqun Zhang. Image fusion network for dual-modal restoration. Inverse Problems and Imaging, 2021, 15 (6) : 1409-1419. doi: 10.3934/ipi.2021067
References:
[1]

E. J. Candes, J. Romberg and T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inform. Theory, 52 (2004), 489-509.  doi: 10.1109/TIT.2005.862083.

[2]

K. Dabov, A. Foi, V. Katkovnik and K. Egiazarian, Image denoising by sparse 3D transform-domain collaborative filtering, IEEE Trans. Image Process, 16 (2007), 2080-2095.  doi: 10.1109/TIP.2007.901238.

[3]

L. Deng, M. Feng and X. Tai, The fusion of panchromatic and multispectral remote sensing images via tensor-based sparse modeling and hyper-Laplacian prior, Inform Fusion, 52 (2019), 76-89. 

[4]

S. Fujieda, K. Takayama and T. Hachisuka, Wavelet Convolutional Neural Networks, CoRR, 2018.

[5]

H. Huang, R. He, Z. Sun and T. Tan, Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution, 2017 IEEE International Conference on Computer Vision (ICCV), (2017), 1698–1706. doi: 10.1109/ICCV.2017.187.

[6]

C. M. Hyun, H. P. Kim, S. M. Lee, S. C. Lee and J. K. Seo, Deep learning for undersampled MRI reconstruction, Phys. Med. Biol., 63 (2018).

[7]

T. B. Kai, M. Uecker and J. Frahm, Undersampled radial MRI with multiple coils. iterative image reconstruction using a total variation constraint, Magn. Reson. Med., 57 (2007), 1086-1098. 

[8]

D. P. Kingma and J. L. Ba, ADAM: A method for stochastic optimization, In Int Conf on Learning Representations, 2015.

[9]

H. Li, B. S. Manjunath and S. K. Mitra, Multisensor image fusion using the wavelet transform, Graphical Models and Image Processing, 57 (1995), 235-245.  doi: 10.1006/gmip.1995.1022.

[10]

L. Li, B. Wang and G. Wang, A self-adaptive mask-enhanced dual-dictionary learning method for MRI-CT image reconstruction, In Nuclear Science Symposium & Medical Imaging Conf, 2016.

[11]

P. Liu, H. Zhang, K. Zhang, L. Lin and W. Zuo, Multi-level wavelet-cnn for image restoration, In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018. doi: 10.1109/CVPRW.2018.00121.

[12]

G. Pajares and J. M. D. L. Cruz, A wavelet-based image fusion tutorial, Pattern Recogn, 37 (2004), 1855-1872. 

[13]

Y. Quan, H. Ji and Z. Shen, Data-driven multi-scale non-local wavelet frame construction and image recovery, J. Sci. Comput., 63 (2015), 307-329.  doi: 10.1007/s10915-014-9893-2.

[14]

R. Singh, M. Vatsa and A. Noore, Multimodal medical image fusion using redundant discrete wavelet transform, InInt. Conf. on Advances in Pattern Recognition, (2009), 232–235. doi: 10.1109/ICAPR.2009.97.

[15]

D. W. Townsend, Multimodality imaging of structure and function, Phys. Med. Biol., 53 (2008), 1-39.  doi: 10.1088/0031-9155/53/4/R01.

[16]

L. Xiang, Y. Chen, W. Chang, Y. Zhan, W. Lin, Q. Wang and D. Shen, Ultra-fast t2-weighted mr reconstruction using complementary t1-weighted information, International Conference on Medical Image Computing and Computer-Assisted Intervention, 11070 (2018), 215-223.  doi: 10.1007/978-3-030-00928-1_25.

[17]

J. C. Ye, Y. Han and E. Cha, Deep convolutional framelets: A general deep learning for inverse problems, SIAM J. Imaginig Sci., 11 (2017), 991-1048.  doi: 10.1137/17M1141771.

[18]

R. Yin, T. Gao, Y. M. Lu and I. Daubechies, A tale of two bases: Local-nonlocal regularization on image patches with convolution framelets, SIAM J. Imaginig Sci., 10 (2017), 711-750.  doi: 10.1137/16M1091447.

[19]

K. Zhang, W. Zuo, Y. Chen, D. Meng and L. Zhang, Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising, IEEE Transactions on Image Processing, 26 (2017), 3142-3155.  doi: 10.1109/TIP.2017.2662206.

[20]

Y. Zhang and X. Zhang, PET-MRI joint reconstruction with common edge weighted total variation regularization, Inverse Probl., 34 (2018), 76-89.  doi: 10.1088/1361-6420/aabce9.

show all references

References:
[1]

E. J. Candes, J. Romberg and T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inform. Theory, 52 (2004), 489-509.  doi: 10.1109/TIT.2005.862083.

[2]

K. Dabov, A. Foi, V. Katkovnik and K. Egiazarian, Image denoising by sparse 3D transform-domain collaborative filtering, IEEE Trans. Image Process, 16 (2007), 2080-2095.  doi: 10.1109/TIP.2007.901238.

[3]

L. Deng, M. Feng and X. Tai, The fusion of panchromatic and multispectral remote sensing images via tensor-based sparse modeling and hyper-Laplacian prior, Inform Fusion, 52 (2019), 76-89. 

[4]

S. Fujieda, K. Takayama and T. Hachisuka, Wavelet Convolutional Neural Networks, CoRR, 2018.

[5]

H. Huang, R. He, Z. Sun and T. Tan, Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution, 2017 IEEE International Conference on Computer Vision (ICCV), (2017), 1698–1706. doi: 10.1109/ICCV.2017.187.

[6]

C. M. Hyun, H. P. Kim, S. M. Lee, S. C. Lee and J. K. Seo, Deep learning for undersampled MRI reconstruction, Phys. Med. Biol., 63 (2018).

[7]

T. B. Kai, M. Uecker and J. Frahm, Undersampled radial MRI with multiple coils. iterative image reconstruction using a total variation constraint, Magn. Reson. Med., 57 (2007), 1086-1098. 

[8]

D. P. Kingma and J. L. Ba, ADAM: A method for stochastic optimization, In Int Conf on Learning Representations, 2015.

[9]

H. Li, B. S. Manjunath and S. K. Mitra, Multisensor image fusion using the wavelet transform, Graphical Models and Image Processing, 57 (1995), 235-245.  doi: 10.1006/gmip.1995.1022.

[10]

L. Li, B. Wang and G. Wang, A self-adaptive mask-enhanced dual-dictionary learning method for MRI-CT image reconstruction, In Nuclear Science Symposium & Medical Imaging Conf, 2016.

[11]

P. Liu, H. Zhang, K. Zhang, L. Lin and W. Zuo, Multi-level wavelet-cnn for image restoration, In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018. doi: 10.1109/CVPRW.2018.00121.

[12]

G. Pajares and J. M. D. L. Cruz, A wavelet-based image fusion tutorial, Pattern Recogn, 37 (2004), 1855-1872. 

[13]

Y. Quan, H. Ji and Z. Shen, Data-driven multi-scale non-local wavelet frame construction and image recovery, J. Sci. Comput., 63 (2015), 307-329.  doi: 10.1007/s10915-014-9893-2.

[14]

R. Singh, M. Vatsa and A. Noore, Multimodal medical image fusion using redundant discrete wavelet transform, InInt. Conf. on Advances in Pattern Recognition, (2009), 232–235. doi: 10.1109/ICAPR.2009.97.

[15]

D. W. Townsend, Multimodality imaging of structure and function, Phys. Med. Biol., 53 (2008), 1-39.  doi: 10.1088/0031-9155/53/4/R01.

[16]

L. Xiang, Y. Chen, W. Chang, Y. Zhan, W. Lin, Q. Wang and D. Shen, Ultra-fast t2-weighted mr reconstruction using complementary t1-weighted information, International Conference on Medical Image Computing and Computer-Assisted Intervention, 11070 (2018), 215-223.  doi: 10.1007/978-3-030-00928-1_25.

[17]

J. C. Ye, Y. Han and E. Cha, Deep convolutional framelets: A general deep learning for inverse problems, SIAM J. Imaginig Sci., 11 (2017), 991-1048.  doi: 10.1137/17M1141771.

[18]

R. Yin, T. Gao, Y. M. Lu and I. Daubechies, A tale of two bases: Local-nonlocal regularization on image patches with convolution framelets, SIAM J. Imaginig Sci., 10 (2017), 711-750.  doi: 10.1137/16M1091447.

[19]

K. Zhang, W. Zuo, Y. Chen, D. Meng and L. Zhang, Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising, IEEE Transactions on Image Processing, 26 (2017), 3142-3155.  doi: 10.1109/TIP.2017.2662206.

[20]

Y. Zhang and X. Zhang, PET-MRI joint reconstruction with common edge weighted total variation regularization, Inverse Probl., 34 (2018), 76-89.  doi: 10.1088/1361-6420/aabce9.

Figure 1.  Proposed network architecture: An observed image $ \mathbf{f} $ and a reference image $ \mathbf{f}_\mathrm{ref} $ are passed as inputs into an encoding network consisting of a cascade of decomposition stages. The outputs of these stages are then passed as the input for a symmetric decoding stage which combines the information from $ \mathbf{f} $ and $ \mathbf{f}_\mathrm{ref} $. The structures labeled by $ \Psi_i $, $ \Psi_i^{'} $, and $ \Psi_i^{†} $ are convolutional layers representing decomposition and composition using learned kernels, while the wavelet transform layers represent decomposition and composition by a set of fixed kernels. The fusion of coefficients indicated by $ \bigoplus $ is multiplied by separated $ w $ and $ w^{'} $, and then performed with a multi-layer convolutional neural network (CNN) H, both of which need to be learned
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 2.  Network inputs for three different denoising scenarios. In each column, the left panel is the target image to be denoised ($ \mathbf{f} $) and the right column is the corresponding reference image ($ \mathbf{f}_\mathrm{ref} $). (a) Denoising of a noisy T2W image with a high-SNR T1W reference image. (b) Denoising a noisy T2W image with an all-zero reference image. (c) "Denoising'' an all-zero image with a high-SNR T1W reference image. The scenarios in (b) and (c) were evaluated to observe how the network incorporates information from the reference image into the denoising process
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 3.  T2W image denoising results for the scenarios depicted in Fig. 2. Top row: ground truth and denoising results of BM3D, DnCNN, and the proposed network, respectively. Bottom row: magnified results from the region indicated by the red box in the ground truth image in the top row. The PSNR and MSE over the entire image are given below each result
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 4.  T2W image with high noise level denoising results. Top row: noisy T2W image and ground truth. Bottom row: denoising results of BM3D, DnCNN, and the proposed network, respectively
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 5.  Reconstruction of sparsely sampled Fourier data. Top row: ground truth versions of several T2W images. Rows 2-4 show the corresponding reconstruction results for a sampling pattern with uniform undersampling in outer $ k $-space. Rows 5-7 show the corresponding reconstruction results for a sampling pattern with random under sampling. The MSE is given on each result
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 6.  Reconstruction of random sampled Fourier data under different noise levels
Figure Options

Download full-size image

Download as PowerPoint slide

Table 1.  Specific parameters setting for e×periments shown in Fig. 5
Encoding Decoding
1th
layer		 comb
layer		 2th
layer		 comb
layer		 3th
layer		 comb
layer		 4th
layer		 comb
layer		 5th
layer		 comb
layer		 1th
layer		 2th
layer		 3th
layer		 4th
layer		 5th
layer
obj $\begin{array}{*{20}{c}} &5 \times 5 \times \\ &1,25 \end{array} $ $\begin{gathered} 25 \times 2 \\ +3 \times 3 \\ \times 25, \\ 25 \times 3 \end{gathered} $ $\begin{gathered} 5 \times 5 \times \\ 25,50 \end{gathered} $ $\begin{gathered} 50 \times 2 \\ +3 \times 3 \\ \times 50, \\ 50 \times 3 \end{gathered} $ $ \begin{gathered} 5 \times 5 \times \\ 50,100 \end{gathered}$ $\begin{gathered} 100 \times 2 \\ +3 \times 3 \\ \times 100, \\ 100 \times 3 \end{gathered} $ \begin{gathered} 5 \times 5 \times \\ 100, \\ 150 \end{gathered}$ $ $ \begin{gathered} 150 \times 2 \\ +3 \times 3 \\ \times 150, \\ 150 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 150, \\ 200 \end{gathered} $ $ \begin{gathered} 200 \times 2 \\ +3 \times 3 \\ \times 200, \\ 200 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 200, \\ 150 \end{gathered} $ $ \begin{gathered} 5 \times 5 \times \\ 150, \\ 100 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 100,50 \end{gathered} $ $ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &50,25 \end{array}$ $ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &25,1 \end{array}$
ref $ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &1,25 \end{array}$ $ \begin{gathered} 5 \times 5 \times \\ 25,50 \end{gathered}$ $ \begin{gathered} 5 \times 5 \times \\ 50,100 \end{gathered}$ $ \begin{gathered} 5 \times 5 \times \\ 100 , \\ 150 \end{gathered}$ $ \begin{gathered} 5 \times 5 \times \\ 150, \\ 200 \end{gathered}$
Table Options

Download as excel

Encoding Decoding
1th
layer		 comb
layer		 2th
layer		 comb
layer		 3th
layer		 comb
layer		 4th
layer		 comb
layer		 5th
layer		 comb
layer		 1th
layer		 2th
layer		 3th
layer		 4th
layer		 5th
layer
obj $\begin{array}{*{20}{c}} &5 \times 5 \times \\ &1,25 \end{array} $ $\begin{gathered} 25 \times 2 \\ +3 \times 3 \\ \times 25, \\ 25 \times 3 \end{gathered} $ $\begin{gathered} 5 \times 5 \times \\ 25,50 \end{gathered} $ $\begin{gathered} 50 \times 2 \\ +3 \times 3 \\ \times 50, \\ 50 \times 3 \end{gathered} $ $ \begin{gathered} 5 \times 5 \times \\ 50,100 \end{gathered}$ $\begin{gathered} 100 \times 2 \\ +3 \times 3 \\ \times 100, \\ 100 \times 3 \end{gathered} $ \begin{gathered} 5 \times 5 \times \\ 100, \\ 150 \end{gathered}$ $ $ \begin{gathered} 150 \times 2 \\ +3 \times 3 \\ \times 150, \\ 150 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 150, \\ 200 \end{gathered} $ $ \begin{gathered} 200 \times 2 \\ +3 \times 3 \\ \times 200, \\ 200 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 200, \\ 150 \end{gathered} $ $ \begin{gathered} 5 \times 5 \times \\ 150, \\ 100 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 100,50 \end{gathered} $ $ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &50,25 \end{array}$ $ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &25,1 \end{array}$
ref $ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &1,25 \end{array}$ $ \begin{gathered} 5 \times 5 \times \\ 25,50 \end{gathered}$ $ \begin{gathered} 5 \times 5 \times \\ 50,100 \end{gathered}$ $ \begin{gathered} 5 \times 5 \times \\ 100 , \\ 150 \end{gathered}$ $ \begin{gathered} 5 \times 5 \times \\ 150, \\ 200 \end{gathered}$
Table 2.  The average MSE of several methods under different noise levels
Noise level E-WTV DnCNN Proposed
0.01 0.000956 0.000703 0.000657
0.05 0.001600 0.000989 0.000926
0.1 0.002700 0.004997 0.001533
Table Options

Download as excel

Noise level E-WTV DnCNN Proposed
0.01 0.000956 0.000703 0.000657
0.05 0.001600 0.000989 0.000926
0.1 0.002700 0.004997 0.001533
[1]

Nicolas Lermé, François Malgouyres, Dominique Hamoir, Emmanuelle Thouin. Bayesian image restoration for mosaic active imaging. Inverse Problems and Imaging, 2014, 8 (3) : 733-760. doi: 10.3934/ipi.2014.8.733
[2]

Jiangchuan Fan, Xinyu Guo, Jianjun Du, Weiliang Wen, Xianju Lu, Brahmani Louiza. Analysis of the clustering fusion algorithm for multi-band color image. Discrete and Continuous Dynamical Systems - S, 2019, 12 (4&5) : 1233-1249. doi: 10.3934/dcdss.2019085
[3]

Yunmei Chen, Jiangli Shi, Murali Rao, Jin-Seop Lee. Deformable multi-modal image registration by maximizing Rényi's statistical dependence measure. Inverse Problems and Imaging, 2015, 9 (1) : 79-103. doi: 10.3934/ipi.2015.9.79
[4]

Mehdi Bastani, Davod Khojasteh Salkuyeh. On the GSOR iteration method for image restoration. Numerical Algebra, Control and Optimization, 2021, 11 (1) : 27-43. doi: 10.3934/naco.2020013
[5]

Eugene Kashdan, Svetlana Bunimovich-Mendrazitsky. Multi-scale model of bladder cancer development. Conference Publications, 2011, 2011 (Special) : 803-812. doi: 10.3934/proc.2011.2011.803
[6]

Amir Averbuch, Pekka Neittaanmäki, Valery Zheludev. Periodic spline-based frames for image restoration. Inverse Problems and Imaging, 2015, 9 (3) : 661-707. doi: 10.3934/ipi.2015.9.661
[7]

Ruiqiang He, Xiangchu Feng, Xiaolong Zhu, Hua Huang, Bingzhe Wei. RWRM: Residual Wasserstein regularization model for image restoration. Inverse Problems and Imaging, 2021, 15 (6) : 1307-1332. doi: 10.3934/ipi.2020069
[8]

Ke Chen, Yiqiu Dong, Michael Hintermüller. A nonlinear multigrid solver with line Gauss-Seidel-semismooth-Newton smoother for the Fenchel pre-dual in total variation based image restoration. Inverse Problems and Imaging, 2011, 5 (2) : 323-339. doi: 10.3934/ipi.2011.5.323
[9]

Yong Zheng Ong, Haizhao Yang. Generative imaging and image processing via generative encoder. Inverse Problems and Imaging, 2022, 16 (3) : 525-545. doi: 10.3934/ipi.2021060
[10]

Thomas Blanc, Mihaï Bostan. Multi-scale analysis for highly anisotropic parabolic problems. Discrete and Continuous Dynamical Systems - B, 2020, 25 (1) : 335-399. doi: 10.3934/dcdsb.2019186
[11]

Michel Potier-Ferry, Foudil Mohri, Fan Xu, Noureddine Damil, Bouazza Braikat, Khadija Mhada, Heng Hu, Qun Huang, Saeid Nezamabadi. Cellular instabilities analyzed by multi-scale Fourier series: A review. Discrete and Continuous Dynamical Systems - S, 2016, 9 (2) : 585-597. doi: 10.3934/dcdss.2016013
[12]

Xiaoman Liu, Jijun Liu. Image restoration from noisy incomplete frequency data by alternative iteration scheme. Inverse Problems and Imaging, 2020, 14 (4) : 583-606. doi: 10.3934/ipi.2020027
[13]

Alina Toma, Bruno Sixou, Françoise Peyrin. Iterative choice of the optimal regularization parameter in TV image restoration. Inverse Problems and Imaging, 2015, 9 (4) : 1171-1191. doi: 10.3934/ipi.2015.9.1171
[14]

Jing Xu, Xue-Cheng Tai, Li-Lian Wang. A two-level domain decomposition method for image restoration. Inverse Problems and Imaging, 2010, 4 (3) : 523-545. doi: 10.3934/ipi.2010.4.523
[15]

Bartomeu Coll, Joan Duran, Catalina Sbert. Half-linear regularization for nonconvex image restoration models. Inverse Problems and Imaging, 2015, 9 (2) : 337-370. doi: 10.3934/ipi.2015.9.337
[16]

Eitan Tadmor, Prashant Athavale. Multiscale image representation using novel integro-differential equations. Inverse Problems and Imaging, 2009, 3 (4) : 693-710. doi: 10.3934/ipi.2009.3.693
[17]

Thierry Cazenave, Flávio Dickstein, Fred B. Weissler. Multi-scale multi-profile global solutions of parabolic equations in $\mathbb{R}^N $. Discrete and Continuous Dynamical Systems - S, 2012, 5 (3) : 449-472. doi: 10.3934/dcdss.2012.5.449
[18]

Emiliano Cristiani, Elisa Iacomini. An interface-free multi-scale multi-order model for traffic flow. Discrete and Continuous Dynamical Systems - B, 2019, 24 (11) : 6189-6207. doi: 10.3934/dcdsb.2019135
[19]

Rongliang Chen, Jizu Huang, Xiao-Chuan Cai. A parallel domain decomposition algorithm for large scale image denoising. Inverse Problems and Imaging, 2019, 13 (6) : 1259-1282. doi: 10.3934/ipi.2019055
[20]

Thomas Y. Hou, Pengfei Liu. Optimal local multi-scale basis functions for linear elliptic equations with rough coefficients. Discrete and Continuous Dynamical Systems, 2016, 36 (8) : 4451-4476. doi: 10.3934/dcds.2016.36.4451

2021 Impact Factor: 1.483

Tools

Metrics

  • PDF downloads (152)
  • HTML views (239)
  • Cited by (0)

Other articles
by authors

[Back to Top]

Copyright © 2022 American Institute of Mathematical Sciences | Privacy Policy