A nonlinear fourth-order PDE for multi-frame image super-resolution enhancement

Amine Laghrib; Abdelkrim Chakib; Aissam Hadri; Abdelilah Hakim

doi:10.3934/dcdsb.2019188

Article Contents

2020, Volume 25, Issue 1: 415-442. Doi: 10.3934/dcdsb.2019188

This issue Previous Article Bi-center problem and Hopf cyclicity of a Cubic Liénard system Next Article On the forward dynamical behavior of nonautonomous systems

A nonlinear fourth-order PDE for multi-frame image super-resolution enhancement

1.
LMA FST Béni-Mellal, Université Sultan Moulay Slimane, Morocco
2.
Faculté Polydisciplinaire Ouarzazate, Morocco
3.
LAMAI, FST Marrakech, Université Cadi Ayyad, Morocco

^* Corresponding author: A.laghrib
^* Corresponding author: A.laghrib

Received: July 31, 2018

Revised: January 31, 2019

Early access: September 2019

Published: January 2020

Abstract Full Text(HTML) Figure(12) / Table(4) Related Papers Cited by

Abstract

The multiframe super-resolution (SR) techniques are considered as one of the active research fields. More precisely, the construction of the desired high resolution (HR) image with less artifacts in the SR models, which are always ill-posed problems, requires a great care. In this paper, we propose a new fourth-order equation based on a diffusive tensor that takes the benefit from the diffusion model of Perona-Malik in the flat regions and the Weickert model near boundaries with a high diffusion order. As a result, the proposed SR approach can efficiently preserve image features such as corner and texture much better with less blur near edges. The existence and uniqueness of the proposed partial differential equation (PDE) are also demonstrated in an appropriate functional space. Finally, the given experimental results show the effectiveness of the proposed PDE compared to some competitive methods in both visually and quantitatively.

Keywords:

Mathematics Subject Classification: Primary: 68U10; Secondary: 94A08.

Citation:

Full Text(HTML)

Figure 1. The original seven images

Download: Full-size image PowerPoint slide

Figure 2. The First six LR images used for the super-resolution process

Download: Full-size image PowerPoint slide

Figure 3. Comparisons of different SR methods (Build image)

Download: Full-size image PowerPoint slide

Figure 4. Comparisons of different SR methods (Lion image)

Download: Full-size image PowerPoint slide

Figure 5. Comparisons of different SR methods (Penguin image)

Download: Full-size image PowerPoint slide

Figure 6. Comparisons of different SR methods (Man image)

Download: Full-size image PowerPoint slide

Figure 7. Comparisons of different SR methods (Surf image)

Download: Full-size image PowerPoint slide

Figure 8. Comparisons of different SR methods (Zebra image)

Download: Full-size image PowerPoint slide

Figure 9. Comparisons of different SR methods (Barbara image)

Download: Full-size image PowerPoint slide

Figure 10. Comparisons of different SR methods (Book image)

Download: Full-size image PowerPoint slide

Figure 11. Comparisons of different SR methods (Disk image)

Download: Full-size image PowerPoint slide

Figure 12. Comparisons of different SR methods (Paint image)

Download: Full-size image PowerPoint slide

Table 1. The PSNR table

Image	PSNR
	$ BTV $ reg. [17]	TV reg. [39]	Adaptive reg. [67]	GDP SR [73]	The PDE in [15]	Our Method
Build	27.00	28.10	28.93	28.50	29.66	$ \mathbf{31.08} $
Lion	26.90	26.95	27.42	27.88	28.33	$ \mathbf{29.51} $
Penguin	31.77	31.50	31.85	32.09	33.00	$ \mathbf{34.02} $
Man	29.55	29.03	29.80	30.40	31.10	$ \mathbf{32.55} $
Surf	30.01	30.05	30.88	30.91	31.44	$ \mathbf{32.12} $
Zebra	29.96	29.92	30.61	30.84	31.22	$ \mathbf{32.18} $

| Show Table

DownLoad: CSV

Table 2. The SSIM table

Image	SSIM
	$ BTV $ reg. [17]	TV reg. [39]	Adaptive reg. [67]	GDP SR [73]	The PDE in [15]	Our Method
Build	0.786	0.783	0.796	0.802	0.812	$ \mathbf{0.830} $
Lion	0.805	0.807	0.837	0.845	0.866	$ \mathbf{0.884} $
Penguin	0.848	0.833	0.846	0.860	0.888	$ \mathbf{0.900} $
Man	0.776	0.772	0.800	0.819	0.836	$ \mathbf{0.874} $
Surf	0.802	0.789	0.816	0.840	0.855	$ \mathbf{0.872} $
Zebra	0.804	0.802	0.811	0.823	0.859	$ \mathbf{0.890} $

| Show Table

DownLoad: CSV

Table 3. The IFC table

Image	IFC
	$ BTV $ reg. [17]	TV reg. [39]	Adaptive reg. [67]	GDP SR [73]	The PDE in [15]	Our Method
Build	1.712	1.700	1.820	1.844	1.901	$ \mathbf{1.980} $
Lion	1.777	1.770	1.778	1.830	1.860	$ \mathbf{1.889} $
Penguin	1.825	1.802	1.933	1.930	1.964	$ \mathbf{2.111} $
Man	1.790	1.768	1.900	1.893	1.992	$ \mathbf{2.155} $
Surf	1.788	1.785	1.801	1.825	1.885	$ \mathbf{2.090} $
Zebra	1.800	1.802	1.863	1.860	1.933	$ \mathbf{2.200} $

| Show Table

DownLoad: CSV

Table 4. CPU times (in seconds) of different super-resolution methods and the proposed method when the magnification factor is $ 4 $

Image	Size	SR algorithms
		$ BTV $ reg. [17]	TV reg. [39]	Adaptive reg. [67]	GDP SR [73]	The PDE in [15]	Our Method
Build	$ 472\times 312 $	9.84	8.26	12.02	9.88	24.96	26.44
Lion	$ 472\times 312 $	6.82	13.64	6.62	7.34	6.61	52.60
Penguin	$ 472\times 312 $	8.31	14.48	7.16	7.62	8.08	48.33
Man	$ 312\times 472 $	10.66	9.75	13.92	9.84	25.93	26.10
Surf	$ 504\times 504 $	11.05	11.18	14.22	10.86	25.80	25.93
Zebra	$ 472\times 312 $	9.86	9.14	12.22	10.32	20.24	22.02
Barbara	$ 512\times 512 $	10.87	9.96	13.88	10.66	26.60	27.10
Average time		13.77	20.06	12.33	14.10	14.66	58.68

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	Super-Resolution Imaging, Digital Imaging and Computer Vision, CRC Press, 2010.
[2]	L. Afraites, A. Atlas, F. Karami and D. Meskine, Some class of parabolic systems applied to image processing, Discrete Contin. Dyn. Syst. Ser B, 21 2016), 1671–1687. doi: 10.3934/dcdsb.2016017.
[3]	A. Aftabizadeh, Existence and uniqueness theorems for fourth-order boundary value problems, Journal of Mathematical Analysis and Applications, 116 (1986), 415-426. doi: 10.1016/S0022-247X(86)80006-3.
[4]	H. Attouch, G. Buttazzo and G. Michaille, Variational Analysis in Sobolev and BV Spaces: Applications to PDEs and Optimization, volume 17., Siam, 2014. doi: 10.1137/1.9781611973488.
[5]	S. Baker and T. Kanade, Limits on super-resolution and how to break them, IEEE Transactions on Pattern Analysis and Machine Intelligence, 24 (2002), 1167-1183. doi: 10.1109/CVPR.2000.854852.
[6]	E. Beretta, M. Bertsch and R. Dal Passo, Nonnegative solutions of a fourth-order nonlinear degenerate parabolic equation, Archive for Rational Mechanics and Analysis, 129 (1995), 175-200. doi: 10.1007/BF00379920.
[7]	F. Bernis and A. Friedman, Higher order nonlinear degenerate parabolic equations, Journal of Differential Equations, 83 (1990), 179-206. doi: 10.1016/0022-0396(90)90074-Y.
[8]	S. Borman and R. L. Stevenson, Super-resolution from image sequences-a review, In Circuits and Systems, 1998. Proceedings. 1998 Midwest Symposium on, pages 374–378. IEEE, 1998. doi: 10.1109/MWSCAS.1998.759509.
[9]	H. Brezis, Functional Analysis, Sobolev Spaces and Partial Differential Equations, Universitext. Springer, New York, 2011.
[10]	F. E. Browder, A new generalization of the schauder fixed point theorem, Mathematische Annalen, 174 (1967), 285-290. doi: 10.1007/BF01364275.
[11]	A. Buades, B. Coll and J.-M. Morel, A non-local algorithm for image denoising, In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, IEEE, 2 (2005), 60–65.
[12]	J. Chaparova, Existence and numerical approximations of periodic solutions of semilinear fourth-order differential equations, Journal of Mathematical Analysis and Applications, 273 (2002), 121-136. doi: 10.1016/S0022-247X(02)00216-0.
[13]	J. Chen, J. Nunez-Yanez and A. Achim, Video super-resolution using generalized gaussian markov random fields, IEEE Signal Processing Letters, 19 (2012), 63-66. doi: 10.1109/LSP.2011.2178595.
[14]	W. Dong, L. Zhang, G. Shi and X. Wu, Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization, IEEE Transactions on Image Processing, 20 (2011), 1838-1857. doi: 10.1109/TIP.2011.2108306.
[15]	I. El Mourabit, M. El Rhabi, A. Hakim, A. Laghrib and E. Moreau, A new denoising model for multi-frame super-resolution image reconstruction, Signal Processing, 132 (2017), 51-65. doi: 10.1016/j.sigpro.2016.09.014.
[16]	S. Farsiu, M. Elad and P. Milanfar, Multiframe demosaicing and super-resolution of color images, IEEE transactions on image processing, 15 (2006), 141-159. doi: 10.1109/TIP.2005.860336.
[17]	S. Farsiu, M. D. Robinson, M. Elad and P. Milanfar, Fast and robust multiframe super resolution, IEEE Transactions on Image Processing, 13 (2004), 1327-1344. doi: 10.1109/TIP.2004.834669.
[18]	M. Fernández-Suárez and A. Y. Ting, Fluorescent probes for super-resolution imaging in living cells, Nature Reviews. Molecular Cell Biology, 9 (2008), 929.
[19]	M. Gao and S. Qin, High performance super-resolution reconstruction of multi-frame degraded images with local weighted anisotropy and successive regularization., Optik-International Journal for Light and Electron Optics, 126 (2015), 4219-4227. doi: 10.1016/j.ijleo.2015.08.119.
[20]	H. Greenspan, Super-resolution in medical imaging, The Computer Journal, 52 (2009), 43-63. doi: 10.1093/comjnl/bxm075.
[21]	H. Greenspan, G. Oz, N. Kiryati and S. Peled, MRI inter-slice reconstruction using super-resolution, International Conference on Medical Image Computing and Computer-Assisted Intervention, 2208 (2001), 1204-1206. doi: 10.1007/3-540-45468-3_164.
[22]	M. R. Hajiaboli, An anisotropic fourth-order diffusion filter for image noise removal, International Journal of Computer Vision, 92 (2011), 177-191. doi: 10.1007/s11263-010-0330-1.
[23]	G. Han and Z. Xu, Multiple solutions of some nonlinear fourth-order beam equations, Nonlinear Analysis: Theory, Methods & Applications, 68 (2008), 3646-3656. doi: 10.1016/j.na.2007.04.007.
[24]	Y. He, K.-H. Yap, L. Chen and L.-P. Chau, Blind super-resolution image reconstruction using a maximum a posteriori estimation, In Image Processing, 2006 IEEE International Conference on, IEEE, 2006, 1729–1732. doi: 10.1109/ICIP.2006.312715.
[25]	Y. He, K.-H. Yap, L. Chen and L.-P. Chau, A nonlinear least square technique for simultaneous image registration and super-resolution, IEEE Transactions on Image Processing, 16 (2007), 2830-2841. doi: 10.1109/TIP.2007.908074.
[26]	T. Hermosilla, E. Bermejo, A. Balaguer and L. A. Ruiz, Non-linear fourth-order image interpolation for subpixel edge detection and localization, Image and Vision Computing, 26 (2008), 1240-1248. doi: 10.1016/j.imavis.2008.02.012.
[27]	D. Holland, D. Boyd and P. Marshall, Updating topographic mapping in great britain using imagery from high-resolution satellite sensors, ISPRS Journal of Photogrammetry and Remote Sensing, 60 (2006), 212-223.
[28]	A. Kanemura, S.-I. Maeda and S. Ishii, Superresolution with compound markov random fields via the variational em algorithm, Neural Networks, 22 (2000), 1025-1034.
[29]	A. Laghrib, A. Ben-Loghfyry, A. Hadri and A. Hakim, A nonconvex fractional order variational model for multi-frame image super-resolution, Signal Processing: Image Communication, 67 (2018), 1-11. doi: 10.1016/j.image.2018.05.011.
[30]	A. Laghrib, A. Ghazdali, A. Hakim and S. Raghay, A multi-frame super-resolution using diffusion registration and a nonlocal variational image restoration, Computers & Mathematics with Applications, 72 (2016), 2535-2548. doi: 10.1016/j.camwa.2016.09.013.
[31]	A. Laghrib, A. Hakim and S. Raghay, A combined total variation and bilateral filter approach for image robust super resolution, EURASIP Journal on Image and Video Processing, 2015 (2015), 1-10. doi: 10.1186/s13640-015-0075-4.
[32]	A. Laghrib, A. Hakim and S. Raghay, An iterative image super-resolution approach based on bregman distance, Signal Processing: Image Communication, 58 (2017), 24-34. doi: 10.1016/j.image.2017.06.006.
[33]	X. Li, Y. Hu, X. Gao, D. Tao and B. Ning, A multi-frame image super-resolution method, Signal Processing, 90 (2010), 405-414. doi: 10.1016/j.sigpro.2009.05.028.
[34]	Y. Li, Positive solutions of fourth-order boundary value problems with two parameters, Journal of Mathematical Analysis and Applications, 281 (2003), 477-484. doi: 10.1016/S0022-247X(03)00131-8.
[35]	J. Lions, Quelques Méthodes de Résolution des Problemes aux Limites non Linéaires, Dunod Paris, 1969.
[36]	J. Lions and E. Magenes, Problèmes Aux Limites non Homogènes et Applications III, Dunod, 1968.
[37]	B. Maiseli, C. Wu, J. Mei, Q. Liu and H. Gao, A robust super-resolution method with improved high-frequency components estimation and aliasing correction capabilities, Journal of the Franklin Institute, 351 (2014), 513-527. doi: 10.1016/j.jfranklin.2013.09.007.
[38]	B. J. Maiseli, N. Ally and H. Gao, A noise-suppressing and edge-preserving multiframe super-resolution image reconstruction method, Signal Processing: Image Communication, 34 (2015), 1-13.
[39]	A. Marquina and S. J. Osher, Image super-resolution by tv-regularization and bregman iteration, Journal of Scientific Computing, 37 (2008), 367-382. doi: 10.1007/s10915-008-9214-8.
[40]	L. Min, X. Yang and D. Ye, Well-posedness for a fourth order nonlinear equation related to image processing, Nonlinear Analysis: Real World Applications, 17 (2014), 192-202. doi: 10.1016/j.nonrwa.2013.11.005.
[41]	K. Nelson, A. Bhatti and S. Nahavandi, Performance evaluation of multi-frame super-resolution algorithms, In Digital Image Computing Techniques and Applications (DICTA), 2012 International Conference on, IEEE, 2012, 1–8.
[42]	M. K. Ng, H. Shen, E. Y. Lam and L. Zhang, A total variation regularization based super-resolution reconstruction algorithm for digital video, EURASIP Journal on Advances in Signal Processing, 2007 (2007), 074585. doi: 10.1155/2007/74585.
[43]	D. O'Regan, Y. J. Cho and Y. Q. Chen, Topological Degree Theory and Applications, Chapman & Hall/CRC Taylor & Francis Group, 2006. doi: 10.1201/9781420011487.
[44]	S. C. Park, M. K. Park and M. G. Kang, Super-resolution image reconstruction: A technical overview, IEEE Signal Processing Magazine, 20 (2003), 21-36.
[45]	P. Perona and J. Malik, Scale-space and edge detection using anisotropic diffusion, IEEE Transactions on Pattern Analysis and Machine Intelligence, 12 (1990), 629-639. doi: 10.1109/34.56205.
[46]	G. Peyré S. Bougleux and L. Cohen, Non-local regularization of inverse problems, Computer Vision–ECCV 2008, (2008), 57–68.
[47]	M. Protter, M. Elad, H. Takeda and P. Milanfar, Generalizing the nonlocal-means to super-resolution reconstruction, IEEE Transactions on Image Processing, 18 (2009), 36-51. doi: 10.1109/TIP.2008.2008067.
[48]	P. Purkait and B. Chanda, Super resolution image reconstruction through bregman iteration using morphologic regularization, IEEE Transactions on Image Processing, 21 (2012), 4029-4039. doi: 10.1109/TIP.2012.2201492.
[49]	D. Rajan and S. Chaudhuri, An mrf-based approach to generation of super-resolution images from blurred observations, Journal of Mathematical Imaging and Vision, 16 (2002), 5-15. doi: 10.1023/A:1013961817285.
[50]	Z. Ren, C. He and Q. Zhang, Fractional order total variation regularization for image super-resolution, Signal Processing, 93 (2013), 2408-2421. doi: 10.1016/j.sigpro.2013.02.015.
[51]	M. D. Robinson, S. J. Chiu, J. Lo, C. Toth, J. Izatt and S. Farsiu, New applications of super-resolution in medical imaging, Super-Resolution Imaging, 2010 (2010), 384-412. doi: 10.1201/9781439819319-13.
[52]	L. I. Rudin, S. Osher and E. Fatemi, Nonlinear total variation based noise removal algorithms, Physica D: Nonlinear Phenomena, 60 (1992), 259-268. doi: 10.1016/0167-2789(92)90242-F.
[53]	P. Sanguansat, K. Thakulsukanant and V. Patanavijit, A robust video super-resolution using a recursive leclerc bayesian approach with an ofom (optical flow observation model), In Advanced Information Networking and Applications Workshops (WAINA), 2012 26th International Conference on, IEEE, 2012,116–121. doi: 10.1109/WAINA.2012.63.
[54]	E. Shechtman, Y. Caspi and M. Irani, Space-time super-resolution, IEEE Transactions on Pattern Analysis and Machine Intelligence, 27 (2005), 531-545. doi: 10.1109/TPAMI.2005.85.
[55]	H. R. Sheikh, A. C. Bovik and G. De Veciana, An information fidelity criterion for image quality assessment using natural scene statistics, IEEE Transactions on Image Processing, 14 (2005), 2117-2128. doi: 10.1109/TIP.2005.859389.
[56]	H. Song, L. Zhang, P. Wang, K. Zhang and X. Li, An adaptive l 1–l 2 hybrid error model to super-resolution, In Image Processing (ICIP), 2010 17th IEEE International Conference on, IEEE, 2010, 2821–2824.
[57]	A. J. Tatem, H. G. Lewis, P. M. Atkinson and M. S. Nixon, Super-resolution target identification from remotely sensed images using a hopfield neural network, IEEE Transactions on Geoscience and Remote Sensing, 39 (2001), 781-796. doi: 10.1109/36.917895.
[58]	R. Y. Tsai and T. S. Huang, Multiframe Image Restoration and Registration, In: Advances in Computer Vision and Image Processing, ed. T.S.Huang. Greenwich, CT, JAI Press, 1984.
[59]	Z. Wang, E. P. Simoncelli and A. C. Bovik, Multiscale structural similarity for image quality assessment, In Signals, Systems and Computers, 2004. Conference Record of the Thirty-Seventh Asilomar Conference on, IEEE, 2 (2003), 1398–1402. doi: 10.1109/ACSSC.2003.1292216.
[60]	J. Weickert, Anisotropic Diffusion in Image Processing, volume 1., Teubner Stuttgart, 1998.
[61]	M. Werlberger, W. Trobin, T. Pock, A. Wedel, D. Cremers and H. Bischof, Anisotropic huber-l1 optical flow, In BMVC, 1 (2009), 3.
[62]	D. Yi and S. Lee, Fourth-order partial differential equations for image enhancement, Applied Mathematics and Computation, 175 (2006), 430-440. doi: 10.1016/j.amc.2005.07.043.
[63]	Y.-L. You and M. Kaveh, Fourth-order partial differential equations for noise removal, IEEE Transactions on Image Processing, 9 (2000), 1723-1730. doi: 10.1109/83.869184.
[64]	Y.-L. You, W. Xu, A. Tannenbaum and M. Kaveh, Behavioral analysis of anisotropic diffusion in image processing, IEEE Transactions on Image Processing, 5 (1996), 1539-1553.
[65]	Q. Yuan, L. Zhang and H. Shen, Multiframe super-resolution employing a spatially weighted total variation model, IEEE Transactions on Circuits and Systems for Video Technology, 22 (2012), 379-392. doi: 10.1109/TCSVT.2011.2163447.
[66]	W. Yuanji, L. Jianhua, L. Yi, F. Yao, and J. Qinzhong., Image quality evaluation based on image weighted separating block peak signal to noise ratio., In Neural Networks and Signal Processing, 2003. Proceedings of the 2003 International Conference on, volume 2, pages 994–997. IEEE, 2003.
[67]	W. Zeng, X. Lu and S. Fei, Image super-resolution employing a spatial adaptive prior model, Neurocomputing, 162 (2015), 218-233. doi: 10.1016/j.neucom.2015.03.049.
[68]	X. Zeng and L. Yang, A robust multiframe super-resolution algorithm based on half-quadratic estimation with modified btv regularization, Digital Signal Processing, 23 (2013), 98-109. doi: 10.1016/j.dsp.2012.06.013.
[69]	H. Zhang, L. Zhang and H. Shen, A super-resolution reconstruction algorithm for hyperspectral images, Signal Processing, 92 (2012), 2082-2096. doi: 10.1016/j.sigpro.2012.01.020.
[70]	L. Zhang, H. Zhang, H. Shen and P. Li, A super-resolution reconstruction algorithm for surveillance images, Signal Processing, 90 (2010), 848-859. doi: 10.1016/j.sigpro.2009.09.002.
[71]	X. Zhang and L. Liu, Positive solutions of fourth-order four-point boundary value problems with p-laplacian operator, Journal of Mathematical Analysis and Applications, 336 (2007), 1414-1423. doi: 10.1016/j.jmaa.2007.03.015.
[72]	X. Zhang and W. Ye, An adaptive fourth-order partial differential equation for image denoising, Computers & Mathematics with Applications, 74 (2017), 2529-2545. doi: 10.1016/j.camwa.2017.07.036.
[73]	S. Zhao, H. Liang and M. Sarem, A generalized detail-preserving super-resolution method, Signal Processing, 120 (2016), 156-173. doi: 10.1016/j.sigpro.2015.09.006.