# American Institute of Mathematical Sciences

January  2021, 17(1): 427-445. doi: 10.3934/jimo.2019119

## Calibration of a 3D laser rangefinder and a camera based on optimization solution

 a. School of Control Science and Engineering, Dalian University of Technology, Dalian 116024, China b. School of Mathematical Sciences, Dalian University of Technology, Dalian 116024, China c. Information Center, Ministry of Water Resources of China, Beijing 100032, China

* Corresponding author: Lei Wang (Email: wanglei@dlut.edu.cn)

Received  March 2019 Revised  April 2019 Published  January 2021 Early access  September 2019

The calibration of a 3D laser rangefinder (LRF) and a camera is a key technique in the field of computer vision and intelligent robots. This paper proposes a new method for the calibration of a 3D LRF and a camera based on optimization solution. The calibration is achieved by freely moving a checkerboard pattern in front of the camera and the 3D LRF. The images and the 3D point clouds of the checkerboard pattern in various poses are collected by the camera and the 3D LRF respectively. By using the images, the intrinsic parameters and the poses of the checkerboard pattern are obtained. Then, two kinds of geometric constraints, line-to-plane constraints and plane-to-plane constraints, are constructed to solve the extrinsic parameters by linear optimization. Finally, the intrinsic and extrinsic parameters are further refined by global optimization, and are used to compute the geometric mapping relationship between the 3D LRF and the camera. The proposed calibration method is evaluated with both synthetic data and real data. The experimental results show that the proposed calibration method is accurate and robust to noise.

Citation: Yi An, Bo Li, Lei Wang, Chao Zhang, Xiaoli Zhou. Calibration of a 3D laser rangefinder and a camera based on optimization solution. Journal of Industrial and Management Optimization, 2021, 17 (1) : 427-445. doi: 10.3934/jimo.2019119
##### References:
 [1] Y. I. Abdel-Aziz and H. M. Karara, Direct linear transformation into object space coordinates in close-range photogrammetry, in Proceedings of the Symposium on Close-Range Photogrammetry, (1971), 1–18. [2] J.-Y. Bouguet, Camera Calibration Toolbox for Matlab, 2003. [3] J. Briales and J. Gonzalez-Jimenez, A minimal solution for the calibration of a 2D laser-rangefinder and a camera based on scene corners, in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, (2015), 1891–1896. doi: 10.1109/IROS.2015.7353625. [4] J. Y. Fan, On the levenberg-marquardt methods for convex constrained nonlinear equations, Journal of Industrial and Management Optimization, 9 (2013), 227-241.  doi: 10.3934/jimo.2013.9.227. [5] O. D. Faugeras and G. Toscani, Camera calibration for 3D computer vision, in Proceedings of International Workshop on Industrial Application of Machine Vision and Machine Intelligence, (1987), 240–247. [6] A. Geiger, F. Moosmann, Ö. Car and B. Schuster, Automatic camera and range sensor calibration using a single shot, in Proceedings of IEEE International Conference on Robotics and Automation, (2012), 3936–3943. doi: 10.1109/ICRA.2012.6224570. [7] R. Gomez-Ojeda, J. Briales, E. Fernandez-Moral and J. Gonzalez-Jimenez, Extrinsic calibration of a 2D laser-rangefinder and a camera based on scene corners, in Proceedings of IEEE International Conference on Robotics and Automation, (2015), 3611–3616. doi: 10.1109/ICRA.2015.7139700. [8] X. J. Gong, Y. Lin and J. L. Liu, 3D LIDAR-camera extrinsic calibration using an arbitrary trihedron, Sensors, 13 (2013), 1902-1918.  doi: 10.3390/s130201902. [9] J. Heikkila and O. Silven, A four-step camera calibration procedure with implicit image correction, in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (1997), 1106–1112. doi: 10.1109/CVPR.1997.609468. [10] Z. Z. Hu, Y. C. Li, N. Li and B. Zhao, Extrinsic calibration of 2-D laser rangefinder and camera from single shot based on minimal solution, IEEE Transactions on Instrumentation and Measurement, 65 (2016), 915-929.  doi: 10.1109/TIM.2016.2518248. [11] G.-M. Lee, J.-H. Lee and S.-Y. Park, Calibration of VLP-16 Lidar and multi-view cameras using a ball for 360 degree 3D color map acquisition, in Proceedings of IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, (2017), 64–69. [12] Z. W. Liu, D. M. Lu, W. X. Qian, G. H. Gu, K. Ren, J. Zhang and X. F. Kong, Calibration of a single-point laser range finder and a camera, Optical and Quantum Electronics, 50 (2018), 447. doi: 10.1007/s11082-018-1702-y. [13] T. Melen, Geometrical modelling and calibration of video cameras for underwater navigation, Psychotherapeut, 60 (1994), 351-352. [14] J. J. Moré, The Levenberg-Marquardt algorithm: Implementation and theory, LNumerical Analysis, Lecture Notes in Math., Springer, Berlin, 630 (1978), 105–116. [15] H. Rushmeier, J. Gomes, F. Giordano, H. E. Shishiny, K. Magerlein and F. Bernardini, Design and use of an in-museum system for artifact capture, in Proceedings of Conference on Computer Vision and Pattern Recognition Workshop, (2003), p8. doi: 10.1109/CVPRW.2003.10005. [16] A. R. F. Sergio, V. Fremont and P. Bonnifait, Extrinsic calibration between a multi-layer Lidar and a camera, in Proceedings of IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, (2008), 214–219. [17] R. Tsai, A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses, IEEE Journal on Robotics and Automation, 3 (1987), 323-344.  doi: 10.1109/JRA.1987.1087109. [18] F. Vasconcelos, J. P. Barreto and U. Nunes, A minimal solution for the extrinsic calibration of a camera and a laser-rangefinder, IEEE Transactions on Pattern Analysis and Machine Intelligence, 34 (2012), 2097-2107.  doi: 10.1109/TPAMI.2012.18. [19] L. Wang, F. C. Wu and Z. Y. Hu, Multi-camera calibration with one-dimensional object under general motions, in Proceedings of IEEE International Conference on Computer Vision, (2007), 1–7. doi: 10.1109/ICCV.2007.4408994. [20] X. H. Ying, G. W. Wang, X. Mei, S. Yang, J. P. Rong and H. B. Zha, A direct method for the extrinsic calibration of a camera and a line scan LIDAR, in Proceedings of IEEE International Conference Mechatronics and Automation, (2014), 571–576. doi: 10.1109/ICMA.2014.6885760. [21] Q. Zhang and R. Pless, Extrinsic calibration of a camera and laser range finder (improves camera calibration), in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, (2004), 2301–2306. [22] Z. Zhang, A flexible new technique for camera calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 (2000), 1330-1334.  doi: 10.1109/34.888718. [23] Z. Y. Zhang, Camera calibration with one-dimensional objects, Computer Vision-ECCV 2002, (2002), 161–174. doi: 10.1007/3-540-47979-1_11. [24] L. P. Zhou, A new minimal solution for the extrinsic calibration of a 2D LIDAR and a camera using three plane-line correspondences, IEEE Sensors Joural, 14 (2014), 442-454.  doi: 10.1109/JSEN.2013.2284789. [25] Y. Zhuang, F. Yan and H. S. Hu, Automatic extrinsic self-calibration for fusing data from monocular vision and 3-D laser scanner, IEEE Transactions on Instrumentation and Measurement, 63 (2014), 1874-1876.  doi: 10.1109/TIM.2014.2307731.

show all references

##### References:
 [1] Y. I. Abdel-Aziz and H. M. Karara, Direct linear transformation into object space coordinates in close-range photogrammetry, in Proceedings of the Symposium on Close-Range Photogrammetry, (1971), 1–18. [2] J.-Y. Bouguet, Camera Calibration Toolbox for Matlab, 2003. [3] J. Briales and J. Gonzalez-Jimenez, A minimal solution for the calibration of a 2D laser-rangefinder and a camera based on scene corners, in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, (2015), 1891–1896. doi: 10.1109/IROS.2015.7353625. [4] J. Y. Fan, On the levenberg-marquardt methods for convex constrained nonlinear equations, Journal of Industrial and Management Optimization, 9 (2013), 227-241.  doi: 10.3934/jimo.2013.9.227. [5] O. D. Faugeras and G. Toscani, Camera calibration for 3D computer vision, in Proceedings of International Workshop on Industrial Application of Machine Vision and Machine Intelligence, (1987), 240–247. [6] A. Geiger, F. Moosmann, Ö. Car and B. Schuster, Automatic camera and range sensor calibration using a single shot, in Proceedings of IEEE International Conference on Robotics and Automation, (2012), 3936–3943. doi: 10.1109/ICRA.2012.6224570. [7] R. Gomez-Ojeda, J. Briales, E. Fernandez-Moral and J. Gonzalez-Jimenez, Extrinsic calibration of a 2D laser-rangefinder and a camera based on scene corners, in Proceedings of IEEE International Conference on Robotics and Automation, (2015), 3611–3616. doi: 10.1109/ICRA.2015.7139700. [8] X. J. Gong, Y. Lin and J. L. Liu, 3D LIDAR-camera extrinsic calibration using an arbitrary trihedron, Sensors, 13 (2013), 1902-1918.  doi: 10.3390/s130201902. [9] J. Heikkila and O. Silven, A four-step camera calibration procedure with implicit image correction, in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (1997), 1106–1112. doi: 10.1109/CVPR.1997.609468. [10] Z. Z. Hu, Y. C. Li, N. Li and B. Zhao, Extrinsic calibration of 2-D laser rangefinder and camera from single shot based on minimal solution, IEEE Transactions on Instrumentation and Measurement, 65 (2016), 915-929.  doi: 10.1109/TIM.2016.2518248. [11] G.-M. Lee, J.-H. Lee and S.-Y. Park, Calibration of VLP-16 Lidar and multi-view cameras using a ball for 360 degree 3D color map acquisition, in Proceedings of IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, (2017), 64–69. [12] Z. W. Liu, D. M. Lu, W. X. Qian, G. H. Gu, K. Ren, J. Zhang and X. F. Kong, Calibration of a single-point laser range finder and a camera, Optical and Quantum Electronics, 50 (2018), 447. doi: 10.1007/s11082-018-1702-y. [13] T. Melen, Geometrical modelling and calibration of video cameras for underwater navigation, Psychotherapeut, 60 (1994), 351-352. [14] J. J. Moré, The Levenberg-Marquardt algorithm: Implementation and theory, LNumerical Analysis, Lecture Notes in Math., Springer, Berlin, 630 (1978), 105–116. [15] H. Rushmeier, J. Gomes, F. Giordano, H. E. Shishiny, K. Magerlein and F. Bernardini, Design and use of an in-museum system for artifact capture, in Proceedings of Conference on Computer Vision and Pattern Recognition Workshop, (2003), p8. doi: 10.1109/CVPRW.2003.10005. [16] A. R. F. Sergio, V. Fremont and P. Bonnifait, Extrinsic calibration between a multi-layer Lidar and a camera, in Proceedings of IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, (2008), 214–219. [17] R. Tsai, A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses, IEEE Journal on Robotics and Automation, 3 (1987), 323-344.  doi: 10.1109/JRA.1987.1087109. [18] F. Vasconcelos, J. P. Barreto and U. Nunes, A minimal solution for the extrinsic calibration of a camera and a laser-rangefinder, IEEE Transactions on Pattern Analysis and Machine Intelligence, 34 (2012), 2097-2107.  doi: 10.1109/TPAMI.2012.18. [19] L. Wang, F. C. Wu and Z. Y. Hu, Multi-camera calibration with one-dimensional object under general motions, in Proceedings of IEEE International Conference on Computer Vision, (2007), 1–7. doi: 10.1109/ICCV.2007.4408994. [20] X. H. Ying, G. W. Wang, X. Mei, S. Yang, J. P. Rong and H. B. Zha, A direct method for the extrinsic calibration of a camera and a line scan LIDAR, in Proceedings of IEEE International Conference Mechatronics and Automation, (2014), 571–576. doi: 10.1109/ICMA.2014.6885760. [21] Q. Zhang and R. Pless, Extrinsic calibration of a camera and laser range finder (improves camera calibration), in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, (2004), 2301–2306. [22] Z. Zhang, A flexible new technique for camera calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 (2000), 1330-1334.  doi: 10.1109/34.888718. [23] Z. Y. Zhang, Camera calibration with one-dimensional objects, Computer Vision-ECCV 2002, (2002), 161–174. doi: 10.1007/3-540-47979-1_11. [24] L. P. Zhou, A new minimal solution for the extrinsic calibration of a 2D LIDAR and a camera using three plane-line correspondences, IEEE Sensors Joural, 14 (2014), 442-454.  doi: 10.1109/JSEN.2013.2284789. [25] Y. Zhuang, F. Yan and H. S. Hu, Automatic extrinsic self-calibration for fusing data from monocular vision and 3-D laser scanner, IEEE Transactions on Instrumentation and Measurement, 63 (2014), 1874-1876.  doi: 10.1109/TIM.2014.2307731.
Calibration board and 3D color laser ranging system
Pinhole camera model
Point cloud of the calibration board and its line point clouds
Calibration images
Normal vector $\bar{\mathit{\boldsymbol{n}}}_{j}$ of the calibration board in $j$th pose
Line-to-plane constraint
The plane-to-plane constraint
Synthetic data generation
Calibration results with the increasing number of poses
Calibration results with the increasing noise level
Calibration results with the number of poses 30 and the noise level 10$mm$
3D color point cloud of the room
3D color point cloud of the housing estate
RMS error analysis results (unit: pixel)
 Method Minimum Average Maximum Our Method 1.3571 3.6962 7.2579 Zhang's Method 1.8495 4.7702 8.6392 Ying's Method 1.5946 3.7866 7.4648
 Method Minimum Average Maximum Our Method 1.3571 3.6962 7.2579 Zhang's Method 1.8495 4.7702 8.6392 Ying's Method 1.5946 3.7866 7.4648
 [1] Vianney Perchet. Approachability, regret and calibration: Implications and equivalences. Journal of Dynamics and Games, 2014, 1 (2) : 181-254. doi: 10.3934/jdg.2014.1.181 [2] Wenxiu Gong, Zuoliang Xu. An alternative tree method for calibration of the local volatility. Journal of Industrial and Management Optimization, 2022, 18 (1) : 137-156. doi: 10.3934/jimo.2020146 [3] Laurent Devineau, Pierre-Edouard Arrouy, Paul Bonnefoy, Alexandre Boumezoued. Fast calibration of the Libor market model with stochastic volatility and displaced diffusion. Journal of Industrial and Management Optimization, 2020, 16 (4) : 1699-1729. doi: 10.3934/jimo.2019025 [4] Gabriel Jouan, Anne Cuzol, Valérie Monbet, Goulven Monnier. Gaussian mixture models for clustering and calibration of ensemble weather forecasts. Discrete and Continuous Dynamical Systems - S, 2022  doi: 10.3934/dcdss.2022037 [5] Xiangyu Gao, Yong Sun. A new heuristic algorithm for laser antimissile strategy optimization. Journal of Industrial and Management Optimization, 2012, 8 (2) : 457-468. doi: 10.3934/jimo.2012.8.457 [6] Dmitri E. Kvasov, Yaroslav D. Sergeyev. Univariate geometric Lipschitz global optimization algorithms. Numerical Algebra, Control and Optimization, 2012, 2 (1) : 69-90. doi: 10.3934/naco.2012.2.69 [7] Claudia Lederman, Noemi Wolanski. An optimization problem with volume constraint for an inhomogeneous operator with nonstandard growth. Discrete and Continuous Dynamical Systems, 2021, 41 (6) : 2907-2946. doi: 10.3934/dcds.2020391 [8] Xiaoliang Li, Cong Wang. An optimization problem in heat conduction with volume constraint and double obstacles. Discrete and Continuous Dynamical Systems, 2022  doi: 10.3934/dcds.2022084 [9] Anna Doubova, Enrique Fernández-Cara. Some geometric inverse problems for the linear wave equation. Inverse Problems and Imaging, 2015, 9 (2) : 371-393. doi: 10.3934/ipi.2015.9.371 [10] Marissa Condon, Alfredo Deaño, Arieh Iserles. On systems of differential equations with extrinsic oscillation. Discrete and Continuous Dynamical Systems, 2010, 28 (4) : 1345-1367. doi: 10.3934/dcds.2010.28.1345 [11] Jiang-Xia Nan, Deng-Feng Li. Linear programming technique for solving interval-valued constraint matrix games. Journal of Industrial and Management Optimization, 2014, 10 (4) : 1059-1070. doi: 10.3934/jimo.2014.10.1059 [12] Yanjun Wang, Shisen Liu. Relaxation schemes for the joint linear chance constraint based on probability inequalities. Journal of Industrial and Management Optimization, 2021  doi: 10.3934/jimo.2021132 [13] Lekbir Afraites, Chorouk Masnaoui, Mourad Nachaoui. Shape optimization method for an inverse geometric source problem and stability at critical shape. Discrete and Continuous Dynamical Systems - S, 2022, 15 (1) : 1-21. doi: 10.3934/dcdss.2021006 [14] Víctor León, Bruno Scárdua. A geometric-analytic study of linear differential equations of order two. Electronic Research Archive, 2021, 29 (2) : 2101-2127. doi: 10.3934/era.2020107 [15] Yacine Chitour, Zhenyu Liao, Romain Couillet. A geometric approach of gradient descent algorithms in linear neural networks. Mathematical Control and Related Fields, 2022  doi: 10.3934/mcrf.2022021 [16] Mahdi Karimi, Seyed Jafar Sadjadi. Optimization of a Multi-Item Inventory model for deteriorating items with capacity constraint using dynamic programming. Journal of Industrial and Management Optimization, 2022, 18 (2) : 1145-1160. doi: 10.3934/jimo.2021013 [17] Behrouz Kheirfam. Multi-parametric sensitivity analysis of the constraint matrix in piecewise linear fractional programming. Journal of Industrial and Management Optimization, 2010, 6 (2) : 347-361. doi: 10.3934/jimo.2010.6.347 [18] Cornel Marius Murea, Dan Tiba. Topological optimization and minimal compliance in linear elasticity. Evolution Equations and Control Theory, 2020, 9 (4) : 1115-1131. doi: 10.3934/eect.2020043 [19] David Russell. Structural parameter optimization of linear elastic systems. Communications on Pure and Applied Analysis, 2011, 10 (5) : 1517-1536. doi: 10.3934/cpaa.2011.10.1517 [20] Yibing Lv, Zhongping Wan. Linear bilevel multiobjective optimization problem: Penalty approach. Journal of Industrial and Management Optimization, 2019, 15 (3) : 1213-1223. doi: 10.3934/jimo.2018092

2021 Impact Factor: 1.411