December  2017, 11(6): 1047-1070. doi: 10.3934/ipi.2017048

Accelerated bregman operator splitting with backtracking

1. 

Department of Mathematics, University of Florida, Gainesville, FL 32611, USA

2. 

Department of Mathematical Sciences, Clemson University, Clemson, SC 29634, USA

3. 

Munitions Directorate, Air Force Research Laboratory, Eglin AFB, FL 32542-9998, USA

* Corresponding author: Xianqi Li

Received  November 2016 Revised  June 2017 Published  September 2017

Fund Project: The first author is supported by NSF grant DMS-1319050, NSF/DMS 1719932 and AFOSR/Eglin (FA8651-08-D-0108). The second author was partially supported by UFII fellowship and AFRL Mathematical Modeling and Optimization Institute.

This paper develops two accelerated Bregman Operator Splitting (BOS) algorithms with backtracking for solving regularized large-scale linear inverse problems, where the regularization term may not be smooth. The first algorithm improves the rate of convergence for BOSVS [5] in terms of the smooth component in the objective function by incorporating Nesterov's multi-step acceleration scheme under the assumption that the feasible set is bounded. The second algorithm is capable of dealing with the case where the feasible set is unbounded. Moreover, it allows more aggressive stepsize than that in the first scheme by properly selecting the penalty parameter and jointly updating the acceleration parameter and stepsize. Both algorithms exhibit better practical performance than BOSVS and AADMM [21], while preserve the same accelerated rate of convergence as that for AADMM. The numerical results on total-variation based image reconstruction problems indicate the effectiveness of the proposed algorithms.

Citation: Yunmei Chen, Xianqi Li, Yuyuan Ouyang, Eduardo Pasiliao. Accelerated bregman operator splitting with backtracking. Inverse Problems and Imaging, 2017, 11 (6) : 1047-1070. doi: 10.3934/ipi.2017048
References:
[1]

A. Beck and M. Teboulle, Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems, IEEE Transactions on Image Processing, 18 (2009), 2419-2434.  doi: 10.1109/TIP.2009.2028250.

[2]

K. BlockM. Uecker and J. Frahm, Undersampled radial MRI with multiple coils. Iterative image reconstruction using a total variation constraint, Magnetic Resonance in Medicine, 57 (2007), 1086-1098.  doi: 10.1002/mrm.21236.

[3]

S. BoydN. ParikhE. ChuB. Peleato and J. Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers, Foundations and Trends® in Machine Learning, 3 (2011), 1-122.  doi: 10.1561/2200000016.

[4]

Y. ChenW. HagerF. HuangD. PhanX. Ye and W. Yin, Fast algorithms for image reconstruction with application to partially parallel MR imaging, SIAM Journal on Imaging Sciences, 5 (2012), 90-118.  doi: 10.1137/100792688.

[5]

Y. ChenW. HagerM. Yashtini and X. Ye, Bregman operator splitting with variable stepsize for total variation image reconstruction, Computational Optimization and Applications, 54 (2013), 317-342.  doi: 10.1007/s10589-012-9519-2.

[6]

W. Deng and W. Yin, On the global and linear convergence of the generalized alternating direction method of multipliers, Journal of Scientific Computing, 66 (2012), 1-28.  doi: 10.1007/s10915-015-0048-x.

[7]

D. Donoho, Compressed sensing, IEEE Transactions on Information Theory, 52 (2006), 1289-1306.  doi: 10.1109/TIT.2006.871582.

[8]

D. Gabay and B. Mercier, A dual algorithm for the solution of nonlinear variational problems via finite element approximation, Computers & Mathematics with Applications, 2 (1976), 17-40.  doi: 10.1016/0898-1221(76)90003-1.

[9]

R. Glowinski and A. Marroco, Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problémes de Dirichlet non linéaires, Revue française d'automatique, informatique, recherche opérationnelle. Analyse numérique, 9 (1975), 41-76. 

[10]

D. GoldfarbS. Ma and K. Scheinberg, Fast multiple-splitting algorithms for convex optimization, SIAM Journal on Optimization, 22 (2012), 533-556.  doi: 10.1137/090780705.

[11]

T. GoldsteinB. O'DonoghueS. Setzer and R. Baraniuk, Fast alternating direction optimization methods, SIAM Journal on Imaging Sciences, 7 (2014), 1588-1623.  doi: 10.1137/120896219.

[12]

T. Goldstein and S. Osher, The split Bregman method for L1-regularized problems, SIAM Journal on Imaging Sciences, 2 (2009), 323-343.  doi: 10.1137/080725891.

[13]

W. HagerC. NgoM. Yashtini and H. Zhang, An alternating direction approximate Newton algorithm for ill-conditioned inverse problems with application to parallel MRI, Name of the Journal, 3 (2015), 139-162.  doi: 10.1007/s40305-015-0078-y.

[14]

W. HagerM. Yashtini and H. Zhang, An $\backslash$mathcalO(1/k) convergence rate for the variable stepsize Bregman operator splitting algorithm, SIAM Journal on Numerical Analysis, 54 (2016), 1535-1556.  doi: 10.1137/15100401X.

[15]

M. Hong and Z. Luo, On the linear convergence of the alternating direction method of multipliers, Mathematical Programming, 162 (2017), 165-199, arXiv:1208.3922.  doi: 10.1007/s10107-016-1034-2.

[16]

C. LiW. YinH. Jiang and Y. Zhang, An efficient augmented Lagrangian method with applications to total variation minimization, Computational Optimization and Applications, 56 (2013), 507-530.  doi: 10.1007/s10589-013-9576-1.

[17]

Q. Liu, J. Luo, S. Wang M. Xiao and M. Ye, An augmented Lagrangian multi-scale dictionary learning algorithm EURASIP Journal on Advances in Signal Processing, 2011 (2011), p58. doi: 10.1186/1687-6180-2011-58.

[18]

M. LustigD. Donoho and J. Pauly, Sparse MRI: The application of compressed sensing for rapid MR imaging, Magnetic Resonance in Medicine, 58 (2007), 1182-1195.  doi: 10.1002/mrm.21391.

[19]

R. Monteiro and B. Svaiter, Iteration-complexity of block-decomposition algorithms and the alternating direction method of multipliers, SIAM Journal on Optimization, 23 (2013), 475-507.  doi: 10.1137/110849468.

[20]

Y. Nesterov, Smooth minimization of non-smooth functions, Mathematical Programming, 103 (2005), 127-152.  doi: 10.1007/s10107-004-0552-5.

[21]

Y. OuyangY. ChenG. Lan and E. Pasiliao, An accelerated linearized alternating direction method of multipliers, SIAM Journal on Imaging Sciences, 8 (2015), 644-681.  doi: 10.1137/14095697X.

[22]

K. PruessmannM. WeigerP. Börnert and P. Boesiger, Advances in sensitivity encoding with arbitrary k-space trajectories, Magnetic Resonance in Medicine, 46 (2001), 638-651. 

[23]

K. ScheinbergD. Goldfarb and X. Bai, Fast first-order methods for composite convex optimization with backtracking, Foundations of Computational Mathematics, 14 (2014), 389-417.  doi: 10.1007/s10208-014-9189-9.

[24]

L. Shepp and B. Logan, The Fourier reconstruction of a head section, IEEE Transactions on Nuclear Science, 21 (1974), 21-43.  doi: 10.1109/TNS.1974.6499235.

[25]

Y. WangJ. YangW. Yin and Y. Zhang A new alternating minimization algorithm for total variation image reconstruction, A new alternating minimization algorithm for total variation image reconstruction, SIAM Journal on Imaging Sciences, 1 (2008), 248-272.  doi: 10.1137/080724265.

[26]

B. Wohlberg, Efficient Algorithms for Convolutional Sparse Representations, IEEE Transactions on Image Processing, 25 (2016), 301-315.  doi: 10.1109/TIP.2015.2495260.

[27]

J. YangY. Zhang and W. Yin, A fast alternating direction method for TVL1-L2 signal reconstruction from partial Fourier data, IEEE Journal of Selected Topics in Signal Processing, 4 (2010), 288-297. 

[28]

X. YeY. Chen and F. Huang, Computational acceleration for MR image reconstruction in partially parallel imaging, IEEE Transactions on Medical Imaging, 30 (2011), 1055-1063. 

[29]

X. ZhangM. BurgerX. Bresson and S. Osher, Bregmanized nonlocal regularization for deconvolution and sparse reconstruction, SIAM Journal on Imaging Sciences, 3 (2010), 253-276.  doi: 10.1137/090746379.

[30]

X. ZhangM. Burger and S. Osher, A unified primal-dual algorithm framework based on Bregman iteration, Journal of Scientific Computing, 46 (2011), 20-46.  doi: 10.1007/s10915-010-9408-8.

[31]

M. Zhu and T. Chan, An efficient primal-dual hybrid gradient algorithm for total variation image restoration, UCLA CAM Report, (2008), 8-34. 

[32]

M. ZhuS. Wright and T. Chan, Duality-based algorithms for total-variation-regularized image restoration, Computational Optimization and Applications, 47 (2010), 377-400.  doi: 10.1007/s10589-008-9225-2.

show all references

References:
[1]

A. Beck and M. Teboulle, Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems, IEEE Transactions on Image Processing, 18 (2009), 2419-2434.  doi: 10.1109/TIP.2009.2028250.

[2]

K. BlockM. Uecker and J. Frahm, Undersampled radial MRI with multiple coils. Iterative image reconstruction using a total variation constraint, Magnetic Resonance in Medicine, 57 (2007), 1086-1098.  doi: 10.1002/mrm.21236.

[3]

S. BoydN. ParikhE. ChuB. Peleato and J. Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers, Foundations and Trends® in Machine Learning, 3 (2011), 1-122.  doi: 10.1561/2200000016.

[4]

Y. ChenW. HagerF. HuangD. PhanX. Ye and W. Yin, Fast algorithms for image reconstruction with application to partially parallel MR imaging, SIAM Journal on Imaging Sciences, 5 (2012), 90-118.  doi: 10.1137/100792688.

[5]

Y. ChenW. HagerM. Yashtini and X. Ye, Bregman operator splitting with variable stepsize for total variation image reconstruction, Computational Optimization and Applications, 54 (2013), 317-342.  doi: 10.1007/s10589-012-9519-2.

[6]

W. Deng and W. Yin, On the global and linear convergence of the generalized alternating direction method of multipliers, Journal of Scientific Computing, 66 (2012), 1-28.  doi: 10.1007/s10915-015-0048-x.

[7]

D. Donoho, Compressed sensing, IEEE Transactions on Information Theory, 52 (2006), 1289-1306.  doi: 10.1109/TIT.2006.871582.

[8]

D. Gabay and B. Mercier, A dual algorithm for the solution of nonlinear variational problems via finite element approximation, Computers & Mathematics with Applications, 2 (1976), 17-40.  doi: 10.1016/0898-1221(76)90003-1.

[9]

R. Glowinski and A. Marroco, Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problémes de Dirichlet non linéaires, Revue française d'automatique, informatique, recherche opérationnelle. Analyse numérique, 9 (1975), 41-76. 

[10]

D. GoldfarbS. Ma and K. Scheinberg, Fast multiple-splitting algorithms for convex optimization, SIAM Journal on Optimization, 22 (2012), 533-556.  doi: 10.1137/090780705.

[11]

T. GoldsteinB. O'DonoghueS. Setzer and R. Baraniuk, Fast alternating direction optimization methods, SIAM Journal on Imaging Sciences, 7 (2014), 1588-1623.  doi: 10.1137/120896219.

[12]

T. Goldstein and S. Osher, The split Bregman method for L1-regularized problems, SIAM Journal on Imaging Sciences, 2 (2009), 323-343.  doi: 10.1137/080725891.

[13]

W. HagerC. NgoM. Yashtini and H. Zhang, An alternating direction approximate Newton algorithm for ill-conditioned inverse problems with application to parallel MRI, Name of the Journal, 3 (2015), 139-162.  doi: 10.1007/s40305-015-0078-y.

[14]

W. HagerM. Yashtini and H. Zhang, An $\backslash$mathcalO(1/k) convergence rate for the variable stepsize Bregman operator splitting algorithm, SIAM Journal on Numerical Analysis, 54 (2016), 1535-1556.  doi: 10.1137/15100401X.

[15]

M. Hong and Z. Luo, On the linear convergence of the alternating direction method of multipliers, Mathematical Programming, 162 (2017), 165-199, arXiv:1208.3922.  doi: 10.1007/s10107-016-1034-2.

[16]

C. LiW. YinH. Jiang and Y. Zhang, An efficient augmented Lagrangian method with applications to total variation minimization, Computational Optimization and Applications, 56 (2013), 507-530.  doi: 10.1007/s10589-013-9576-1.

[17]

Q. Liu, J. Luo, S. Wang M. Xiao and M. Ye, An augmented Lagrangian multi-scale dictionary learning algorithm EURASIP Journal on Advances in Signal Processing, 2011 (2011), p58. doi: 10.1186/1687-6180-2011-58.

[18]

M. LustigD. Donoho and J. Pauly, Sparse MRI: The application of compressed sensing for rapid MR imaging, Magnetic Resonance in Medicine, 58 (2007), 1182-1195.  doi: 10.1002/mrm.21391.

[19]

R. Monteiro and B. Svaiter, Iteration-complexity of block-decomposition algorithms and the alternating direction method of multipliers, SIAM Journal on Optimization, 23 (2013), 475-507.  doi: 10.1137/110849468.

[20]

Y. Nesterov, Smooth minimization of non-smooth functions, Mathematical Programming, 103 (2005), 127-152.  doi: 10.1007/s10107-004-0552-5.

[21]

Y. OuyangY. ChenG. Lan and E. Pasiliao, An accelerated linearized alternating direction method of multipliers, SIAM Journal on Imaging Sciences, 8 (2015), 644-681.  doi: 10.1137/14095697X.

[22]

K. PruessmannM. WeigerP. Börnert and P. Boesiger, Advances in sensitivity encoding with arbitrary k-space trajectories, Magnetic Resonance in Medicine, 46 (2001), 638-651. 

[23]

K. ScheinbergD. Goldfarb and X. Bai, Fast first-order methods for composite convex optimization with backtracking, Foundations of Computational Mathematics, 14 (2014), 389-417.  doi: 10.1007/s10208-014-9189-9.

[24]

L. Shepp and B. Logan, The Fourier reconstruction of a head section, IEEE Transactions on Nuclear Science, 21 (1974), 21-43.  doi: 10.1109/TNS.1974.6499235.

[25]

Y. WangJ. YangW. Yin and Y. Zhang A new alternating minimization algorithm for total variation image reconstruction, A new alternating minimization algorithm for total variation image reconstruction, SIAM Journal on Imaging Sciences, 1 (2008), 248-272.  doi: 10.1137/080724265.

[26]

B. Wohlberg, Efficient Algorithms for Convolutional Sparse Representations, IEEE Transactions on Image Processing, 25 (2016), 301-315.  doi: 10.1109/TIP.2015.2495260.

[27]

J. YangY. Zhang and W. Yin, A fast alternating direction method for TVL1-L2 signal reconstruction from partial Fourier data, IEEE Journal of Selected Topics in Signal Processing, 4 (2010), 288-297. 

[28]

X. YeY. Chen and F. Huang, Computational acceleration for MR image reconstruction in partially parallel imaging, IEEE Transactions on Medical Imaging, 30 (2011), 1055-1063. 

[29]

X. ZhangM. BurgerX. Bresson and S. Osher, Bregmanized nonlocal regularization for deconvolution and sparse reconstruction, SIAM Journal on Imaging Sciences, 3 (2010), 253-276.  doi: 10.1137/090746379.

[30]

X. ZhangM. Burger and S. Osher, A unified primal-dual algorithm framework based on Bregman iteration, Journal of Scientific Computing, 46 (2011), 20-46.  doi: 10.1007/s10915-010-9408-8.

[31]

M. Zhu and T. Chan, An efficient primal-dual hybrid gradient algorithm for total variation image restoration, UCLA CAM Report, (2008), 8-34. 

[32]

M. ZhuS. Wright and T. Chan, Duality-based algorithms for total-variation-regularized image restoration, Computational Optimization and Applications, 47 (2010), 377-400.  doi: 10.1007/s10589-008-9225-2.

Figure 1.  The objective function values and relative error vs. CPU time for first instance
Figure 2.  The objective function values and relative error vs.CPU time for second instance
Figure 3.  Sensitivity maps of data1
Figure 4.  Left: True image; Right: Cartesian mask for data 1
Figure 5.  Comparison of 400 iterations of BOSVS, TVAL3, ALADMML, and ABOSVS-Ⅱ in partially parallel image reconstruction (Top) and their differences with true image (Bottom) using data1. From left to right: BOSVS, TVAL3, ALADMML, and ABOSVS-Ⅱ
Figure 6.  Sensitivity maps of data2
Figure 7.  Left: True image; Right: Cartesian mask for data 2
Figure 8.  Comparison of 400 iterations of BOSVS, TVAL3, ALADMML, and ABOSVS-Ⅱ in partially parallel image reconstruction (Top) and their differences with true image (Bottom) using data2. From left to right: BOSVS, TVAL3, ALADMML, and ABOSVS-Ⅱ.
Table 1.  Parameter settings for ABOSVS-I and ABOSVS-Ⅱ in (4.1). Note that $\rho$ and $\tau$ are only used in the first instance
ParametersασηminCβkρτ
Values102||A||2/101001/k11
ParametersασηminCβkρτ
Values102||A||2/101001/k11
Table 2.  Parameter settings for ABOSVS-Ⅱ in 4.2
ParametersασηminC
Values10-10n2||A||2/5100
ParametersασηminC
Values10-10n2||A||2/5100
Table 3.  Comparison of objective function value, relative error, and CPU time in seconds using data1
AlgorithmsObjective valueRelative errorCPU
BOSVS15.468410.05962528.0
TVAL356.72630.050228.6
ALADMML15.468100.049966.5
ABOSVS-Ⅱ15.445350.047949.7
AlgorithmsObjective valueRelative errorCPU
BOSVS15.468410.05962528.0
TVAL356.72630.050228.6
ALADMML15.468100.049966.5
ABOSVS-Ⅱ15.445350.047949.7
Table 4.  Comparison of objective function value, relative error, and CPU time in seconds using data2
AlgorithmsObjective valueRelative errorCPU
BOSVS5007e+20.06467448.4
TVAL31704e+30.050681.2
ALADMML5.006e+20.0499180.0
ABOSVS-Ⅱ4.970e+20.0479144.7
AlgorithmsObjective valueRelative errorCPU
BOSVS5007e+20.06467448.4
TVAL31704e+30.050681.2
ALADMML5.006e+20.0499180.0
ABOSVS-Ⅱ4.970e+20.0479144.7
[1]

Leyu Hu, Wenxing Zhang, Xingju Cai, Deren Han. A parallel operator splitting algorithm for solving constrained total-variation retinex. Inverse Problems and Imaging, 2020, 14 (6) : 1135-1156. doi: 10.3934/ipi.2020058

[2]

Yunhai Xiao, Junfeng Yang, Xiaoming Yuan. Alternating algorithms for total variation image reconstruction from random projections. Inverse Problems and Imaging, 2012, 6 (3) : 547-563. doi: 10.3934/ipi.2012.6.547

[3]

Yan Gu, Nobuo Yamashita. A proximal ADMM with the Broyden family for convex optimization problems. Journal of Industrial and Management Optimization, 2021, 17 (5) : 2715-2732. doi: 10.3934/jimo.2020091

[4]

Huan Gao, Yu-Hong Dai, Xiao-Jiao Tong. Barzilai-Borwein-like methods for the extreme eigenvalue problem. Journal of Industrial and Management Optimization, 2015, 11 (3) : 999-1019. doi: 10.3934/jimo.2015.11.999

[5]

Fan Jiang, Zhongming Wu, Xingju Cai. Generalized ADMM with optimal indefinite proximal term for linearly constrained convex optimization. Journal of Industrial and Management Optimization, 2020, 16 (2) : 835-856. doi: 10.3934/jimo.2018181

[6]

Jin-Zan Liu, Xin-Wei Liu. A dual Bregman proximal gradient method for relatively-strongly convex optimization. Numerical Algebra, Control and Optimization, 2021  doi: 10.3934/naco.2021028

[7]

Mujibur Rahman Chowdhury, Jun Zhang, Jing Qin, Yifei Lou. Poisson image denoising based on fractional-order total variation. Inverse Problems and Imaging, 2020, 14 (1) : 77-96. doi: 10.3934/ipi.2019064

[8]

Haijuan Hu, Jacques Froment, Baoyan Wang, Xiequan Fan. Spatial-Frequency domain nonlocal total variation for image denoising. Inverse Problems and Imaging, 2020, 14 (6) : 1157-1184. doi: 10.3934/ipi.2020059

[9]

Juan C. Moreno, V. B. Surya Prasath, João C. Neves. Color image processing by vectorial total variation with gradient channels coupling. Inverse Problems and Imaging, 2016, 10 (2) : 461-497. doi: 10.3934/ipi.2016008

[10]

Zhengmeng Jin, Chen Zhou, Michael K. Ng. A coupled total variation model with curvature driven for image colorization. Inverse Problems and Imaging, 2016, 10 (4) : 1037-1055. doi: 10.3934/ipi.2016031

[11]

Sudeb Majee, Subit K. Jain, Rajendra K. Ray, Ananta K. Majee. A fuzzy edge detector driven telegraph total variation model for image despeckling. Inverse Problems and Imaging, 2022, 16 (2) : 367-396. doi: 10.3934/ipi.2021054

[12]

Hai Huyen Dam, Siow Yong Low, Sven Nordholm. Two-level optimization approach with accelerated proximal gradient for objective measures in sparse speech reconstruction. Journal of Industrial and Management Optimization, 2021  doi: 10.3934/jimo.2021131

[13]

Baoli Shi, Zhi-Feng Pang, Jing Xu. Image segmentation based on the hybrid total variation model and the K-means clustering strategy. Inverse Problems and Imaging, 2016, 10 (3) : 807-828. doi: 10.3934/ipi.2016022

[14]

Xavier Bresson, Tony F. Chan. Fast dual minimization of the vectorial total variation norm and applications to color image processing. Inverse Problems and Imaging, 2008, 2 (4) : 455-484. doi: 10.3934/ipi.2008.2.455

[15]

Ke Chen, Yiqiu Dong, Michael Hintermüller. A nonlinear multigrid solver with line Gauss-Seidel-semismooth-Newton smoother for the Fenchel pre-dual in total variation based image restoration. Inverse Problems and Imaging, 2011, 5 (2) : 323-339. doi: 10.3934/ipi.2011.5.323

[16]

Xiaoqun Zhang, Tony F. Chan. Wavelet inpainting by nonlocal total variation. Inverse Problems and Imaging, 2010, 4 (1) : 191-210. doi: 10.3934/ipi.2010.4.191

[17]

Gang Li, Xiaoqi Yang, Yuying Zhou. Stable strong and total parametrized dualities for DC optimization problems in locally convex spaces. Journal of Industrial and Management Optimization, 2013, 9 (3) : 671-687. doi: 10.3934/jimo.2013.9.671

[18]

Lateef Olakunle Jolaoso, Maggie Aphane. Bregman subgradient extragradient method with monotone self-adjustment stepsize for solving pseudo-monotone variational inequalities and fixed point problems. Journal of Industrial and Management Optimization, 2022, 18 (2) : 773-794. doi: 10.3934/jimo.2020178

[19]

Rinaldo M. Colombo, Francesca Monti. Solutions with large total variation to nonconservative hyperbolic systems. Communications on Pure and Applied Analysis, 2010, 9 (1) : 47-60. doi: 10.3934/cpaa.2010.9.47

[20]

Zehui Jia, Xue Gao, Xingju Cai, Deren Han. The convergence rate analysis of the symmetric ADMM for the nonconvex separable optimization problems. Journal of Industrial and Management Optimization, 2021, 17 (4) : 1943-1971. doi: 10.3934/jimo.2020053

2020 Impact Factor: 1.639

Metrics

  • PDF downloads (136)
  • HTML views (234)
  • Cited by (1)

[Back to Top]