
-
Previous Article
An adaptive total variational despeckling model based on gray level indicator frame
- IPI Home
- This Issue
-
Next Article
Image retinex based on the nonconvex TV-type regularization
Image fusion network for dual-modal restoration
1. | School of Statistics and Mathematics, Shanghai Lixin University of Accounting and Finance, Shanghai, 201209, China |
2. | School of Biomedical Engineering, Shanghai Jiao Tong University, China |
3. | Department of Electrical and Computer Engineering, Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign, USA |
4. | School of Mathematical Sciences, MOE-LSC, Shanghai Jiao Tong University, China, Institute of Natural Sciences, Shanghai Jiao Tong University, China |
In recent years multi-modal data processing methods have gained considerable research interest as technological advancements in imaging, computing, and data storage have made the collection of redundant, multi-modal data more commonplace. In this work we present an image restoration method tailored for scenarios where pre-existing, high-quality images from different modalities or contrasts are available in addition to the target image. Our method is based on a novel network architecture which combines the benefits of traditional multi-scale signal representation, such as wavelets, with more recent concepts from data fusion methods. Results from numerical simulations in which T1-weighted MRI images are used to restore noisy and undersampled T2-weighted images demonstrate that the proposed network successfully utilizes information from high-quality reference images to improve the restoration quality of the target image beyond that of existing popular methods.
References:
[1] |
E. J. Candes, J. Romberg and T. Tao,
Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inform. Theory, 52 (2004), 489-509.
doi: 10.1109/TIT.2005.862083. |
[2] |
K. Dabov, A. Foi, V. Katkovnik and K. Egiazarian,
Image denoising by sparse 3D transform-domain collaborative filtering, IEEE Trans. Image Process, 16 (2007), 2080-2095.
doi: 10.1109/TIP.2007.901238. |
[3] |
L. Deng, M. Feng and X. Tai,
The fusion of panchromatic and multispectral remote sensing images via tensor-based sparse modeling and hyper-Laplacian prior, Inform Fusion, 52 (2019), 76-89.
|
[4] |
S. Fujieda, K. Takayama and T. Hachisuka, Wavelet Convolutional Neural Networks, CoRR, 2018. |
[5] |
H. Huang, R. He, Z. Sun and T. Tan, Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution, 2017 IEEE International Conference on Computer Vision (ICCV), (2017), 1698–1706.
doi: 10.1109/ICCV.2017.187. |
[6] |
C. M. Hyun, H. P. Kim, S. M. Lee, S. C. Lee and J. K. Seo, Deep learning for undersampled MRI reconstruction, Phys. Med. Biol., 63 (2018). |
[7] |
T. B. Kai, M. Uecker and J. Frahm,
Undersampled radial MRI with multiple coils. iterative image reconstruction using a total variation constraint, Magn. Reson. Med., 57 (2007), 1086-1098.
|
[8] |
D. P. Kingma and J. L. Ba, ADAM: A method for stochastic optimization, In Int Conf on Learning Representations, 2015. |
[9] |
H. Li, B. S. Manjunath and S. K. Mitra,
Multisensor image fusion using the wavelet transform, Graphical Models and Image Processing, 57 (1995), 235-245.
doi: 10.1006/gmip.1995.1022. |
[10] |
L. Li, B. Wang and G. Wang, A self-adaptive mask-enhanced dual-dictionary learning method for MRI-CT image reconstruction, In Nuclear Science Symposium & Medical Imaging Conf, 2016. |
[11] |
P. Liu, H. Zhang, K. Zhang, L. Lin and W. Zuo, Multi-level wavelet-cnn for image restoration, In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018.
doi: 10.1109/CVPRW.2018.00121. |
[12] |
G. Pajares and J. M. D. L. Cruz,
A wavelet-based image fusion tutorial, Pattern Recogn, 37 (2004), 1855-1872.
|
[13] |
Y. Quan, H. Ji and Z. Shen,
Data-driven multi-scale non-local wavelet frame construction and image recovery, J. Sci. Comput., 63 (2015), 307-329.
doi: 10.1007/s10915-014-9893-2. |
[14] |
R. Singh, M. Vatsa and A. Noore, Multimodal medical image fusion using redundant discrete wavelet transform, InInt. Conf. on Advances in Pattern Recognition, (2009), 232–235.
doi: 10.1109/ICAPR.2009.97. |
[15] |
D. W. Townsend,
Multimodality imaging of structure and function, Phys. Med. Biol., 53 (2008), 1-39.
doi: 10.1088/0031-9155/53/4/R01. |
[16] |
L. Xiang, Y. Chen, W. Chang, Y. Zhan, W. Lin, Q. Wang and D. Shen,
Ultra-fast t2-weighted mr reconstruction using complementary t1-weighted information, International Conference on Medical Image Computing and Computer-Assisted Intervention, 11070 (2018), 215-223.
doi: 10.1007/978-3-030-00928-1_25. |
[17] |
J. C. Ye, Y. Han and E. Cha,
Deep convolutional framelets: A general deep learning for inverse problems, SIAM J. Imaginig Sci., 11 (2017), 991-1048.
doi: 10.1137/17M1141771. |
[18] |
R. Yin, T. Gao, Y. M. Lu and I. Daubechies,
A tale of two bases: Local-nonlocal regularization on image patches with convolution framelets, SIAM J. Imaginig Sci., 10 (2017), 711-750.
doi: 10.1137/16M1091447. |
[19] |
K. Zhang, W. Zuo, Y. Chen, D. Meng and L. Zhang,
Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising, IEEE Transactions on Image Processing, 26 (2017), 3142-3155.
doi: 10.1109/TIP.2017.2662206. |
[20] |
Y. Zhang and X. Zhang,
PET-MRI joint reconstruction with common edge weighted total variation regularization, Inverse Probl., 34 (2018), 76-89.
doi: 10.1088/1361-6420/aabce9. |
show all references
References:
[1] |
E. J. Candes, J. Romberg and T. Tao,
Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inform. Theory, 52 (2004), 489-509.
doi: 10.1109/TIT.2005.862083. |
[2] |
K. Dabov, A. Foi, V. Katkovnik and K. Egiazarian,
Image denoising by sparse 3D transform-domain collaborative filtering, IEEE Trans. Image Process, 16 (2007), 2080-2095.
doi: 10.1109/TIP.2007.901238. |
[3] |
L. Deng, M. Feng and X. Tai,
The fusion of panchromatic and multispectral remote sensing images via tensor-based sparse modeling and hyper-Laplacian prior, Inform Fusion, 52 (2019), 76-89.
|
[4] |
S. Fujieda, K. Takayama and T. Hachisuka, Wavelet Convolutional Neural Networks, CoRR, 2018. |
[5] |
H. Huang, R. He, Z. Sun and T. Tan, Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution, 2017 IEEE International Conference on Computer Vision (ICCV), (2017), 1698–1706.
doi: 10.1109/ICCV.2017.187. |
[6] |
C. M. Hyun, H. P. Kim, S. M. Lee, S. C. Lee and J. K. Seo, Deep learning for undersampled MRI reconstruction, Phys. Med. Biol., 63 (2018). |
[7] |
T. B. Kai, M. Uecker and J. Frahm,
Undersampled radial MRI with multiple coils. iterative image reconstruction using a total variation constraint, Magn. Reson. Med., 57 (2007), 1086-1098.
|
[8] |
D. P. Kingma and J. L. Ba, ADAM: A method for stochastic optimization, In Int Conf on Learning Representations, 2015. |
[9] |
H. Li, B. S. Manjunath and S. K. Mitra,
Multisensor image fusion using the wavelet transform, Graphical Models and Image Processing, 57 (1995), 235-245.
doi: 10.1006/gmip.1995.1022. |
[10] |
L. Li, B. Wang and G. Wang, A self-adaptive mask-enhanced dual-dictionary learning method for MRI-CT image reconstruction, In Nuclear Science Symposium & Medical Imaging Conf, 2016. |
[11] |
P. Liu, H. Zhang, K. Zhang, L. Lin and W. Zuo, Multi-level wavelet-cnn for image restoration, In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018.
doi: 10.1109/CVPRW.2018.00121. |
[12] |
G. Pajares and J. M. D. L. Cruz,
A wavelet-based image fusion tutorial, Pattern Recogn, 37 (2004), 1855-1872.
|
[13] |
Y. Quan, H. Ji and Z. Shen,
Data-driven multi-scale non-local wavelet frame construction and image recovery, J. Sci. Comput., 63 (2015), 307-329.
doi: 10.1007/s10915-014-9893-2. |
[14] |
R. Singh, M. Vatsa and A. Noore, Multimodal medical image fusion using redundant discrete wavelet transform, InInt. Conf. on Advances in Pattern Recognition, (2009), 232–235.
doi: 10.1109/ICAPR.2009.97. |
[15] |
D. W. Townsend,
Multimodality imaging of structure and function, Phys. Med. Biol., 53 (2008), 1-39.
doi: 10.1088/0031-9155/53/4/R01. |
[16] |
L. Xiang, Y. Chen, W. Chang, Y. Zhan, W. Lin, Q. Wang and D. Shen,
Ultra-fast t2-weighted mr reconstruction using complementary t1-weighted information, International Conference on Medical Image Computing and Computer-Assisted Intervention, 11070 (2018), 215-223.
doi: 10.1007/978-3-030-00928-1_25. |
[17] |
J. C. Ye, Y. Han and E. Cha,
Deep convolutional framelets: A general deep learning for inverse problems, SIAM J. Imaginig Sci., 11 (2017), 991-1048.
doi: 10.1137/17M1141771. |
[18] |
R. Yin, T. Gao, Y. M. Lu and I. Daubechies,
A tale of two bases: Local-nonlocal regularization on image patches with convolution framelets, SIAM J. Imaginig Sci., 10 (2017), 711-750.
doi: 10.1137/16M1091447. |
[19] |
K. Zhang, W. Zuo, Y. Chen, D. Meng and L. Zhang,
Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising, IEEE Transactions on Image Processing, 26 (2017), 3142-3155.
doi: 10.1109/TIP.2017.2662206. |
[20] |
Y. Zhang and X. Zhang,
PET-MRI joint reconstruction with common edge weighted total variation regularization, Inverse Probl., 34 (2018), 76-89.
doi: 10.1088/1361-6420/aabce9. |






Encoding | Decoding | ||||||||||||||
1th layer |
comb layer |
2th layer |
comb layer |
3th layer |
comb layer |
4th layer |
comb layer |
5th layer |
comb layer |
1th layer |
2th layer |
3th layer |
4th layer |
5th layer |
|
obj | |||||||||||||||
ref |
Encoding | Decoding | ||||||||||||||
1th layer |
comb layer |
2th layer |
comb layer |
3th layer |
comb layer |
4th layer |
comb layer |
5th layer |
comb layer |
1th layer |
2th layer |
3th layer |
4th layer |
5th layer |
|
obj | |||||||||||||||
ref |
Noise level | E-WTV | DnCNN | Proposed |
0.01 | 0.000956 | 0.000703 | 0.000657 |
0.05 | 0.001600 | 0.000989 | 0.000926 |
0.1 | 0.002700 | 0.004997 | 0.001533 |
Noise level | E-WTV | DnCNN | Proposed |
0.01 | 0.000956 | 0.000703 | 0.000657 |
0.05 | 0.001600 | 0.000989 | 0.000926 |
0.1 | 0.002700 | 0.004997 | 0.001533 |
[1] |
Nicolas Lermé, François Malgouyres, Dominique Hamoir, Emmanuelle Thouin. Bayesian image restoration for mosaic active imaging. Inverse Problems and Imaging, 2014, 8 (3) : 733-760. doi: 10.3934/ipi.2014.8.733 |
[2] |
Jiangchuan Fan, Xinyu Guo, Jianjun Du, Weiliang Wen, Xianju Lu, Brahmani Louiza. Analysis of the clustering fusion algorithm for multi-band color image. Discrete and Continuous Dynamical Systems - S, 2019, 12 (4&5) : 1233-1249. doi: 10.3934/dcdss.2019085 |
[3] |
Yunmei Chen, Jiangli Shi, Murali Rao, Jin-Seop Lee. Deformable multi-modal image registration by maximizing Rényi's statistical dependence measure. Inverse Problems and Imaging, 2015, 9 (1) : 79-103. doi: 10.3934/ipi.2015.9.79 |
[4] |
Mehdi Bastani, Davod Khojasteh Salkuyeh. On the GSOR iteration method for image restoration. Numerical Algebra, Control and Optimization, 2021, 11 (1) : 27-43. doi: 10.3934/naco.2020013 |
[5] |
Eugene Kashdan, Svetlana Bunimovich-Mendrazitsky. Multi-scale model of bladder cancer development. Conference Publications, 2011, 2011 (Special) : 803-812. doi: 10.3934/proc.2011.2011.803 |
[6] |
Amir Averbuch, Pekka Neittaanmäki, Valery Zheludev. Periodic spline-based frames for image restoration. Inverse Problems and Imaging, 2015, 9 (3) : 661-707. doi: 10.3934/ipi.2015.9.661 |
[7] |
Ruiqiang He, Xiangchu Feng, Xiaolong Zhu, Hua Huang, Bingzhe Wei. RWRM: Residual Wasserstein regularization model for image restoration. Inverse Problems and Imaging, 2021, 15 (6) : 1307-1332. doi: 10.3934/ipi.2020069 |
[8] |
Ke Chen, Yiqiu Dong, Michael Hintermüller. A nonlinear multigrid solver with line Gauss-Seidel-semismooth-Newton smoother for the Fenchel pre-dual in total variation based image restoration. Inverse Problems and Imaging, 2011, 5 (2) : 323-339. doi: 10.3934/ipi.2011.5.323 |
[9] |
Yong Zheng Ong, Haizhao Yang. Generative imaging and image processing via generative encoder. Inverse Problems and Imaging, 2022, 16 (3) : 525-545. doi: 10.3934/ipi.2021060 |
[10] |
Thomas Blanc, Mihaï Bostan. Multi-scale analysis for highly anisotropic parabolic problems. Discrete and Continuous Dynamical Systems - B, 2020, 25 (1) : 335-399. doi: 10.3934/dcdsb.2019186 |
[11] |
Michel Potier-Ferry, Foudil Mohri, Fan Xu, Noureddine Damil, Bouazza Braikat, Khadija Mhada, Heng Hu, Qun Huang, Saeid Nezamabadi. Cellular instabilities analyzed by multi-scale Fourier series: A review. Discrete and Continuous Dynamical Systems - S, 2016, 9 (2) : 585-597. doi: 10.3934/dcdss.2016013 |
[12] |
Xiaoman Liu, Jijun Liu. Image restoration from noisy incomplete frequency data by alternative iteration scheme. Inverse Problems and Imaging, 2020, 14 (4) : 583-606. doi: 10.3934/ipi.2020027 |
[13] |
Alina Toma, Bruno Sixou, Françoise Peyrin. Iterative choice of the optimal regularization parameter in TV image restoration. Inverse Problems and Imaging, 2015, 9 (4) : 1171-1191. doi: 10.3934/ipi.2015.9.1171 |
[14] |
Jing Xu, Xue-Cheng Tai, Li-Lian Wang. A two-level domain decomposition method for image restoration. Inverse Problems and Imaging, 2010, 4 (3) : 523-545. doi: 10.3934/ipi.2010.4.523 |
[15] |
Bartomeu Coll, Joan Duran, Catalina Sbert. Half-linear regularization for nonconvex image restoration models. Inverse Problems and Imaging, 2015, 9 (2) : 337-370. doi: 10.3934/ipi.2015.9.337 |
[16] |
Eitan Tadmor, Prashant Athavale. Multiscale image representation using novel integro-differential equations. Inverse Problems and Imaging, 2009, 3 (4) : 693-710. doi: 10.3934/ipi.2009.3.693 |
[17] |
Thierry Cazenave, Flávio Dickstein, Fred B. Weissler. Multi-scale multi-profile global solutions of parabolic equations in $\mathbb{R}^N $. Discrete and Continuous Dynamical Systems - S, 2012, 5 (3) : 449-472. doi: 10.3934/dcdss.2012.5.449 |
[18] |
Emiliano Cristiani, Elisa Iacomini. An interface-free multi-scale multi-order model for traffic flow. Discrete and Continuous Dynamical Systems - B, 2019, 24 (11) : 6189-6207. doi: 10.3934/dcdsb.2019135 |
[19] |
Rongliang Chen, Jizu Huang, Xiao-Chuan Cai. A parallel domain decomposition algorithm for large scale image denoising. Inverse Problems and Imaging, 2019, 13 (6) : 1259-1282. doi: 10.3934/ipi.2019055 |
[20] |
Thomas Y. Hou, Pengfei Liu. Optimal local multi-scale basis functions for linear elliptic equations with rough coefficients. Discrete and Continuous Dynamical Systems, 2016, 36 (8) : 4451-4476. doi: 10.3934/dcds.2016.36.4451 |
2021 Impact Factor: 1.483
Tools
Metrics
Other articles
by authors
[Back to Top]