American Institute of Mathematical Sciences

Advanced
  • Previous Article
    Eliminating other-race effect for multi-ethnic facial expression recognition
  • MFC Home
  • This Issue
  • Next Article
    Comparisons of different methods for balanced data classification under the discrete non-local total variational framework
February  2019, 2(1): 29-41. doi: 10.3934/mfc.2019003

SEMANTIC-RTAB-MAP (SRM): A semantic SLAM system with CNNs on depth images

Mingyuan Mao 1, , Hewei Zhang 1, , Simeng Li 1, and  Baochang Zhang 1,2,,

1. 

Beihang University, Beijing, China
2. 

Shenzhen Academy of Aerospace Technology, Shenzhen, China

* Corresponding author: bczhang@buaa.edu.cn

Published  March 2019

SLAM (simultaneous localization and mapping) system can be implemented based on monocular, RGB-D and stereo cameras. RTAB-MAP is a SLAM system, which can build dense 3D map. In this paper, we present a novel method named SEMANTIC-RTAB-MAP (SRM) to implement a semantic SLAM system based on RTAB-MAP and deep learning. We use YOLOv2 network to detect target objects in 2D images, and then use depth information for precise localization of the targets and finally add semantic information into 3D point clouds. We apply SRM in different scenes, and the results show its higher running speed and accuracy.

Keywords: Semantic SLAM, YOLO, real-time, depth-image, region growing.
Mathematics Subject Classification: Primary: 58F15, 58F17; Secondary: 53C35.
Citation: Mingyuan Mao, Hewei Zhang, Simeng Li, Baochang Zhang. SEMANTIC-RTAB-MAP (SRM): A semantic SLAM system with CNNs on depth images. Mathematical Foundations of Computing, 2019, 2 (1) : 29-41. doi: 10.3934/mfc.2019003
References:
[1]

R. Q. Charles, H. Su, K. Mo and L. J. Guibas, Pointnet: Deep learning on point sets for 3d classification and segmentation, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017). doi: 10.1109/CVPR.2017.16.

[2]

R. Girshick and J. Donahue, Trevor Darrell and Jitendra Malik, Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation, (2013), 580-587.

[3]

K. He, X. Zhang, S. Ren and J. Sun, Deep residual learning for image recognition, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2016), 770-778. doi: 10.1109/CVPR.2016.90.

[4]

M. Labbé and F. Michaud, Long-term online multi-session graph-based splam with memory management, Autonomous Robots, 3 (2017), 1-18. 

[5]

M. Labbe and F. Michaud, Online global loop closure detection for large-scale multi-session graph-based SLAM, IEEE/RSJ International Conference on Intelligent Robots and Systems, (2014), 2661-2666. doi: 10.1109/IROS.2014.6942926.

[6]

M. Labbé and F. Michaud, Appearance-based loop closure detection for online large-scale and long-term operation, IEEE Transactions on Robotics, 29 (2013), 734-745. 

[7]

M. Labbe and F. Michaud, Memory management for real-time appearance-based loop closure detection, IEEE/RSJ International Conference on Intelligent Robots and Systems, (2011), 1271-1276. doi: 10.1109/IROS.2011.6094602.

[8]

X. Li and R. Belaroussi, Semi-dense 3d semantic mapping from monocular slam, 2016.

[9]

W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed and C. Y. Fu, et al, SSD: Single Shot MultiBox Detector. European Conference on Computer Vision, Springer International Publishing, (2016), 21-37.

[10]

J. Mccormac, A. Handa, A. Davison and S. Leutenegger, Semanticfusion: dense 3d semantic mapping with convolutional neural networks, 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017. doi: 10.1109/ICRA.2017.7989538.

[11]

R. Mur-Artal and J. D. Tardós, Probabilistic semi-dense mapping from highly accurate feature-based monocular SLAM, Robotics: Science and Systems, (2015), 1-9. doi: 10.15607/RSS.2015.XI.041.

[12]

N. Otsu, A threshold selection method from gray-level histograms, IEEE Transactions on Systems, Man, and Cybernetics, 9 (1979), 62-66.  doi: 10.1109/TSMC.1979.4310076.

[13]

J. Redmon, S. Divvala, R. Girshick and A. Farhadi, You only look once: Unified, real-time object detection, Computer Vision and Pattern Recognition, (2016), 779-788. doi: 10.1109/CVPR.2016.91.

[14]

J. Redmon and A. Farhadi, YOLO9000: Better, faster, stronger, IEEE Conference on Computer Vision and Pattern Recognition, (2017), 6517-6525. doi: 10.1109/CVPR.2017.690.

[15]

J. Redmon and A. Farhadi, Yolov3: an incremental improvement, 2018.

[16]

S. Ren, K. He, R. Girshick and J. Sun, Faster R-CNN: Towards real-time object detection with region proposal networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 39 (2017), 1137-1149.  doi: 10.1109/TPAMI.2016.2577031.

[17]

N. Sünderhauf, T. T. Pham, Y. Latif, M. Milford and I. Reid, Meaningful maps with object-oriented semantic mapping., Ieee/rsj International Conference on Intelligent Robots and Systems, IEEE, (2017), 5079-5085.

[18]

T. Whelan, S. Leutenegger, R. S. Moreno, B. Glocker and A. Davison, ElasticFusion: Dense SLAM Without A Pose Graph. Robotics: Science and Systems, 2015. doi: 10.15607/RSS.2015.XI.001.

show all references

References:
[1]

R. Q. Charles, H. Su, K. Mo and L. J. Guibas, Pointnet: Deep learning on point sets for 3d classification and segmentation, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017). doi: 10.1109/CVPR.2017.16.

[2]

R. Girshick and J. Donahue, Trevor Darrell and Jitendra Malik, Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation, (2013), 580-587.

[3]

K. He, X. Zhang, S. Ren and J. Sun, Deep residual learning for image recognition, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2016), 770-778. doi: 10.1109/CVPR.2016.90.

[4]

M. Labbé and F. Michaud, Long-term online multi-session graph-based splam with memory management, Autonomous Robots, 3 (2017), 1-18. 

[5]

M. Labbe and F. Michaud, Online global loop closure detection for large-scale multi-session graph-based SLAM, IEEE/RSJ International Conference on Intelligent Robots and Systems, (2014), 2661-2666. doi: 10.1109/IROS.2014.6942926.

[6]

M. Labbé and F. Michaud, Appearance-based loop closure detection for online large-scale and long-term operation, IEEE Transactions on Robotics, 29 (2013), 734-745. 

[7]

M. Labbe and F. Michaud, Memory management for real-time appearance-based loop closure detection, IEEE/RSJ International Conference on Intelligent Robots and Systems, (2011), 1271-1276. doi: 10.1109/IROS.2011.6094602.

[8]

X. Li and R. Belaroussi, Semi-dense 3d semantic mapping from monocular slam, 2016.

[9]

W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed and C. Y. Fu, et al, SSD: Single Shot MultiBox Detector. European Conference on Computer Vision, Springer International Publishing, (2016), 21-37.

[10]

J. Mccormac, A. Handa, A. Davison and S. Leutenegger, Semanticfusion: dense 3d semantic mapping with convolutional neural networks, 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017. doi: 10.1109/ICRA.2017.7989538.

[11]

R. Mur-Artal and J. D. Tardós, Probabilistic semi-dense mapping from highly accurate feature-based monocular SLAM, Robotics: Science and Systems, (2015), 1-9. doi: 10.15607/RSS.2015.XI.041.

[12]

N. Otsu, A threshold selection method from gray-level histograms, IEEE Transactions on Systems, Man, and Cybernetics, 9 (1979), 62-66.  doi: 10.1109/TSMC.1979.4310076.

[13]

J. Redmon, S. Divvala, R. Girshick and A. Farhadi, You only look once: Unified, real-time object detection, Computer Vision and Pattern Recognition, (2016), 779-788. doi: 10.1109/CVPR.2016.91.

[14]

J. Redmon and A. Farhadi, YOLO9000: Better, faster, stronger, IEEE Conference on Computer Vision and Pattern Recognition, (2017), 6517-6525. doi: 10.1109/CVPR.2017.690.

[15]

J. Redmon and A. Farhadi, Yolov3: an incremental improvement, 2018.

[16]

S. Ren, K. He, R. Girshick and J. Sun, Faster R-CNN: Towards real-time object detection with region proposal networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 39 (2017), 1137-1149.  doi: 10.1109/TPAMI.2016.2577031.

[17]

N. Sünderhauf, T. T. Pham, Y. Latif, M. Milford and I. Reid, Meaningful maps with object-oriented semantic mapping., Ieee/rsj International Conference on Intelligent Robots and Systems, IEEE, (2017), 5079-5085.

[18]

T. Whelan, S. Leutenegger, R. S. Moreno, B. Glocker and A. Davison, ElasticFusion: Dense SLAM Without A Pose Graph. Robotics: Science and Systems, 2015. doi: 10.15607/RSS.2015.XI.001.

Figure 1.  Performance of SRM. (First from Left) Original RGB-D image including a label of handbag. (Second from Left) Result of edge detection on the depth image. (Second from Right) Result of region growing. (First from Right) The corresponding 3D point cloud
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 2.  Results by Li et. al. [8]
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 3.  The Structure of Memory Management. [7]
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 4.  YOLOv1 structure. [13]
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 5.  Results by YOLOv2
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 6.  Overview of SRM method
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 7.  Results by YOLOv2
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 8.  The flow chart of our precise localization
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 9.  (Top) Original RGB-D image. (Middle Left) Result of edge detection by Canny operator. (Middle Right) Result of region growing. (Bottom) Semantic RGB-D image
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 10.  The left is the original local point cloud of RTAB-MAP. The right is the corresponding point cloud of SRM. Two refrigerators are recognized by YOLOv2 and painted with green (we assume that green represents refrigerators)
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 11.  (First Column) Original RGB image. (Second Column) Edges of targets extracted in depth image by Canny operator. (Third Column) Corresponding local point clouds. The blue sticks in the point clouds in the first three rows are axes. Because we detected different targets separately, some different targets are painted the same color. When we detect them at the same time, we just need to assign different colors to different classes of objects
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 12.  The left is the original RGB image. The right is the corresponding point cloud. We paint the bottle red, the laptop blue and the handbag green. We don't show Edges of targets extracted in depth image here because SRM process different objects one by one, which means we don't have an image including all of their edges. The handbag is not shown completely in the point cloud because the Kinect2 didn't get the depth data of that area
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 13.  The performance of semi-dense 3D semantic mapping. [8] (Top) Original image. (Bottom) Corresponding point cloud. Red represents buildings. Purple represents cars. Green bounding boxes are added by us. They are not included in the original image
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 14.  The left is the original RGB image. The right is the point cloud
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 15.  The left is the original RGB image. The right is the point cloud
Figure Options

Download full-size image

Download as PowerPoint slide

[1]

Lingshuang Kong, Changjun Yu, Kok Lay Teo, Chunhua Yang. Robust real-time optimization for blending operation of alumina production. Journal of Industrial and Management Optimization, 2017, 13 (3) : 1149-1167. doi: 10.3934/jimo.2016066
[2]

Melody Dodd, Jennifer L. Mueller. A real-time D-bar algorithm for 2-D electrical impedance tomography data. Inverse Problems and Imaging, 2014, 8 (4) : 1013-1031. doi: 10.3934/ipi.2014.8.1013
[3]

Thomas Demoor, Joris Walraevens, Dieter Fiems, Stijn De Vuyst, Herwig Bruneel. Influence of real-time queue capacity on system contents in DiffServ's expedited forwarding per-hop-behavior. Journal of Industrial and Management Optimization, 2010, 6 (3) : 587-602. doi: 10.3934/jimo.2010.6.587
[4]

Xiang-Sheng Wang, Luoyi Zhong. Ebola outbreak in West Africa: real-time estimation and multiple-wave prediction. Mathematical Biosciences & Engineering, 2015, 12 (5) : 1055-1063. doi: 10.3934/mbe.2015.12.1055
[5]

Matthieu Canaud, Lyudmila Mihaylova, Jacques Sau, Nour-Eddin El Faouzi. Probability hypothesis density filtering for real-time traffic state estimation and prediction. Networks and Heterogeneous Media, 2013, 8 (3) : 825-842. doi: 10.3934/nhm.2013.8.825
[6]

Le Thi Hoai An, Tran Duc Quynh, Kondo Hloindo Adjallah. A difference of convex functions algorithm for optimal scheduling and real-time assignment of preventive maintenance jobs on parallel processors. Journal of Industrial and Management Optimization, 2014, 10 (1) : 243-258. doi: 10.3934/jimo.2014.10.243
[7]

Wei Huang, Ka-Fai Cedric Yiu, Henry Y. K. Lau. Semi-definite programming based approaches for real-time tractor localization in port container terminals. Numerical Algebra, Control and Optimization, 2013, 3 (4) : 665-680. doi: 10.3934/naco.2013.3.665
[8]

Yeming Dai, Yan Gao, Hongwei Gao, Hongbo Zhu, Lu Li. A real-time pricing scheme considering load uncertainty and price competition in smart grid market. Journal of Industrial and Management Optimization, 2020, 16 (2) : 777-793. doi: 10.3934/jimo.2018178
[9]

Chengtao Yong, Yan Huo, Chunqiang Hu, Yanfei Lu, Guanlin Jing. A real-time aggregate data publishing scheme with adaptive ω-event differential privacy. Mathematical Foundations of Computing, 2018, 1 (3) : 295-309. doi: 10.3934/mfc.2018014
[10]

Tao Guan, Denghua Zhong, Bingyu Ren, Pu Cheng. Construction schedule optimization for high arch dams based on real-time interactive simulation. Journal of Industrial and Management Optimization, 2015, 11 (4) : 1321-1342. doi: 10.3934/jimo.2015.11.1321
[11]

Xiaoyong Mao, Baoguo Yu, Yingjing Shi, Rui Li. Real-time online trajectory planning and guidance for terminal area energy management of unmanned aerial vehicle. Journal of Industrial and Management Optimization, 2022  doi: 10.3934/jimo.2022026
[12]

Yanxue Yang, Shou-Qiang Du, Yuanyuan Chen. Real-time pricing method for smart grid based on social welfare maximization model. Journal of Industrial and Management Optimization, 2022  doi: 10.3934/jimo.2022039
[13]

Wan Nor Ashikin Wan Ahmad Fatthi, Adibah Shuib, Rosma Mohd Dom. A mixed integer programming model for solving real-time truck-to-door assignment and scheduling problem at cross docking warehouse. Journal of Industrial and Management Optimization, 2016, 12 (2) : 431-447. doi: 10.3934/jimo.2016.12.431
[14]

Susu Zhang, Jiancheng Ni, Lijun Hou, Zili Zhou, Jie Hou, Feng Gao. Global-Affine and Local-Specific Generative Adversarial Network for semantic-guided image generation. Mathematical Foundations of Computing, 2021, 4 (3) : 145-165. doi: 10.3934/mfc.2021009
[15]

Ibrar Hussain, Haider Ali, Muhammad Shahkar Khan, Sijie Niu, Lavdie Rada. Robust region-based active contour models via local statistical similarity and local similarity factor for intensity inhomogeneity and high noise image segmentation. Inverse Problems and Imaging, , () : -. doi: 10.3934/ipi.2022014
[16]

Peng Zhang, Yongquan Zeng, Guotai Chi. Time-consistent multiperiod mean semivariance portfolio selection with the real constraints. Journal of Industrial and Management Optimization, 2021, 17 (4) : 1663-1680. doi: 10.3934/jimo.2020039
[17]

Boran Hu, Zehui Cheng, Zhangbing Zhou. Web services recommendation leveraging semantic similarity computing. Mathematical Foundations of Computing, 2018, 1 (2) : 101-119. doi: 10.3934/mfc.2018006
[18]

Anna Erschler. Iterated identities and iterational depth of groups. Journal of Modern Dynamics, 2015, 9: 257-284. doi: 10.3934/jmd.2015.9.257
[19]

Moulay Rchid Sidi Ammi, Ismail Jamiai. Finite difference and Legendre spectral method for a time-fractional diffusion-convection equation for image restoration. Discrete and Continuous Dynamical Systems - S, 2018, 11 (1) : 103-117. doi: 10.3934/dcdss.2018007
[20]

Simone Fiori. Auto-regressive moving-average discrete-time dynamical systems and autocorrelation functions on real-valued Riemannian matrix manifolds. Discrete and Continuous Dynamical Systems - B, 2014, 19 (9) : 2785-2808. doi: 10.3934/dcdsb.2014.19.2785

 Impact Factor: 

Tools

Metrics

  • PDF downloads (1493)
  • HTML views (2056)
  • Cited by (1)

Other articles
by authors

[Back to Top]

Copyright © 2022 American Institute of Mathematical Sciences | Privacy Policy