Welcome to Journal of Beijing Institute of Technology
Advance Search
Volume 30Issue 2
Jun. 2021
Turn off MathJax
Article Contents
Article Navigation> JOURNAL OF BEIJING INSTITUTE OF TECHNOLOGY> 2021> 30(2): 172-185
Huifang Li, Jianghang Huang, Jingwei Huang, Senchun Chai, Leilei Zhao, Yuanqing Xia. Deep Multimodal Learning and Fusion Based Intelligent Fault Diagnosis Approach[J]. JOURNAL OF BEIJING INSTITUTE OF TECHNOLOGY, 2021, 30(2): 172-185. doi: 10.15918/j.jbit1004-0579.2021.017
Citation: Huifang Li, Jianghang Huang, Jingwei Huang, Senchun Chai, Leilei Zhao, Yuanqing Xia. Deep Multimodal Learning and Fusion Based Intelligent Fault Diagnosis Approach[J].JOURNAL OF BEIJING INSTITUTE OF TECHNOLOGY, 2021, 30(2): 172-185.doi:10.15918/j.jbit1004-0579.2021.017
shu

Deep Multimodal Learning and Fusion Based Intelligent Fault Diagnosis Approach

doi:10.15918/j.jbit1004-0579.2021.017

  • Key Laboratory of Complex System Intelligent Control and Decision, Beijing Institute of Technology, Beijing 100081, China

Funds:This work was supported in part by the National Key Research and Development Program of China (No. 2018YFB1003700) and in part by the National Natural Science Foundation of China (No. 61836001).
More Information
  • Author Bio:

    Huifang Lireceived her B.S. degree in Automatic Control from Nanjing University of Aeronautical and Astronautics, Nanjing, China, in 1988, and M.S. and Ph.D. degrees in Systems Engineering from Xi’an Jiaotong University, Xi’an, China in 1994 and 2000 respectively. She was a postdoctoral researcher in Department of Automation, Tsinghua University, Beijing, China in 2001-2003. She worked as a researcher in Computer Integrated Manufacturing Research Unit, National University of Ireland, Galway in 2007-2008. She was a visiting scholar in University of California, Berkeley, California, United States from May to Aug. 2012, and Research Center of Enterprise Integration and Urban Systems Engineering, University of Toronto, Ontario, Canada from Jun. to Dec. 2016. She is currently an associate professor in School of Automation, Beijing Institute of Technology, Beijing, China. Her research interests are machine learning, industrial fault diagnosis, cloud computing and task scheduling

    Jianghang Huangreceived the B.S. degree from China Agricultural University, Beijing, China, in 2019. He is currently working toward a Master’s degree in Beijing Institute of Technology, Beijing, China. His current research interests include workflow scheduling and reinforcement learning

    Jingwei Huangreceived his B.S. degree in Electrical Engineering and Automation from Beijing Institute of Technology, Beijing, China, in 2020. He is currently working toward a Master’s degree in Electronic Information at Beijing Institute of Technology, Beijing, China. His current research interest is intelligent optimization

    Senchun Chaireceived the B.S. Eng and M.S. Eng degrees in School of Automation from the Beijing Institute of Technology, Beijing, China, in 2001 and 2004, respectively, and the Ph.D. degree and Postdoc fellowship from the School of Electronics, University of Glamorgan, Pontypridd, U.K., in 2007 and 2009, respectively. He has been a researcher with Cranfield University, Cranfield, U.K., from 2009 to 2010, and a visiting scholar with the University of Illinois at Urbana-Champaign, Urbana, IL, USA, from January 2010 to May 2010. He is now a professor with the Beijing Institute of Technology. His research interests include design of Industrial Internet, network security, multi-agent systems, networked control systems, wireless sensor network and aircraft control

    Leilei Zhaoreceived her B.S. degree from North China Electric Power University, Baoding, China, in 2017. She received her Master’s degree from Beijing Institute of Technology, Beijing, China, in 2019. Her current research interests include deep learning and ad recommendation systems

    Yuanqing Xiareceived the M.S. degree in fundamental mathematics from Anhui University, Hefei, China, in 1998, and the Ph.D. degree in control theory and control engineering from the Beijing University of Aeronautics and Astronautics, Beijing, China, in 2001. He was a Post-Doctoral Research Associate with the Institute of Systems Science, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, from 2002 to 2003. From 2004 to 2006, he was with the University of Glamorgan, Pontypridd, U.K., as a Research Fellow. From 2007 to 2008, he was a Guest Professor with Innsbruck Medical University, Innsbruck, Austria. He is currently a Professor with the School of Automation, Beijing Institute of Technology, Beijing. His research interests are cloud control systems, networked control systems, robust control and signal processing, active disturbance rejection control, unmanned system control, and flight control

  • Corresponding author:Huifang Li, Jianghang Huang, Jingwei Huang, Senchun Chai, Leilei Zhao and Yuanqing Xia are with the Key Laboratory of Complex System Intelligent Control and Decision, Beijing Institute of Technology, Beijing 100081, China.
  • Received Date:2021-03-31
  • Rev Recd Date:2021-05-24
  • Accepted Date:2021-05-28
    • Available Online:2021-10-13
  • Publish Date:2021-06-30
    Abstract
  • Industrial Internet of Things (IoT) connecting society and industrial systems represents a tremendous and promising paradigm shift. With IoT, multimodal and heterogeneous data from industrial devices can be easily collected, and further analyzed to discover device maintenance and health related potential knowledge behind. IoT data-based fault diagnosis for industrial devices is very helpful to the sustainability and applicability of an IoT ecosystem. But how to efficiently use and fuse this multimodal heterogeneous data to realize intelligent fault diagnosis is still a challenge. In this paper, a novel Deep Multimodal Learning and Fusion (DMLF) based fault diagnosis method is proposed for addressing heterogeneous data from IoT environments where industrial devices coexist. First, a DMLF model is designed by combining a Convolution Neural Network (CNN) and Stacked Denoising Autoencoder (SDAE) together to capture more comprehensive fault knowledge and extract features from different modal data. Second, these multimodal features are seamlessly integrated at a fusion layer and the resulting fused features are further used to train a classifier for recognizing potential faults. Third, a two-stage training algorithm is proposed by combining supervised pre-training and fine-tuning to simplify the training process for deep structure models. A series of experiments are conducted over multimodal heterogeneous data from a gear device to verify our proposed fault diagnosis method. The experimental results show that our method outperforms the benchmarking ones in fault diagnosis accuracy.
    • FullText(HTML)
  • loading
    • References (36)
  • [1]
    N. Wu, Z. Li, K. Barkaoui, X. Li, T. Murata, and M. Zhou,“IoT-based smart and complex systems: A guest editorial report,” IEEE/CAA Journal of Automatica Sinica, vol. 5, no. 1, pp. 69-73, 2018. doi:10.1109/JAS.2017.7510748
    [2]
    Y. Bai, J. Yang, J. Wang, Y. Zhao, and Q. Li,“Image representation of vibration signals and its application in intelligent compound fault diagnosis in railway vehicle wheelset-axlebox assemblies,” Mechanical Systems and Signal Processing, vol. 152, pp. 107421, 2020.
    [3]
    H. Li, G. Hu, J. Li, and M. Zhou, “Intelligent fault diagnosis for large-scale rotating machines using binarized deep neural networks and random forests,” IEEE Transactions on Automation Science and Engineering, Jan. 2021. DOI: 10.1109/TASE.2020.3048056.
    [4]
    C. Zhang, Y. He, B. Du, L. Yuan, B. Li, and S. Jiang,“Transformer fault diagnosis method using IoT based monitoring system and ensemble machine learning,” Future Generation Computer Systems, vol. 108, pp. 533-545, 2020. doi:10.1016/j.future.2020.03.008
    [5]
    A. Namigtle-Jimenez, R. F. Escobar-Jimenez, J. F. Gomez-Aguilar, C. D. Garcia-Beltran, and A. C. Tellez-Anguiano,“Online ANN-based fault diagnosis implementation using an FPGA: Application in the EFI system of a vehicle,” ISA Transactions, vol. 100, pp. 358-372, 2020. doi:10.1016/j.isatra.2019.11.003
    [6]
    C. He, T. Wu, R. Gu, and H. Qu,“Bearing fault diagnosis based on wavelet packet energy spectrum and SVM,” Journal of Physics: Conference Series, vol. 1684, no. 1, pp. 012135, 2020.
    [7]
    X. Gao, G. Yuan, and M. Zhang, “Fault detection of electric vehicle charging piles based on extreme learning machine algorithm, ” in 4th International Conference on Computing Methodologies and Communication, Erode, India, Mar. 2020, pp. 849-852.
    [8]
    H. Yuan and Z. Chen, “Design and implementation of intelligent fault diagnosis system, ” in 2009 First International Conference on Information Science and Engineering, Nanjing, China, Dec. 2009, pp. 2296-2299.
    [9]
    S. Ma and F. Chu,“Ensemble deep learning-based fault diagnosis of rotor bearing systems,” Computers in Industry, vol. 105, pp. 143-152, 2019. doi:10.1016/j.compind.2018.12.012
    [10]
    B. Liu, Y. Ma, and H. Shi,“A multimodal process fault detection method based on weighted distance neighborhood selection strategy,” Journal of East China University of Science and Technology, vol. 41, no. 2, pp. 192-197, 2015.
    [11]
    X. Li, L. Ping, L. Jiang, and Y. Cao,“Fault diagnosis method of asynchronous motor based on heterogeneous information feature fusion,” Chinese Journal of Scientific Instrument, vol. 34, no. 1, pp. 227-233, 2013.
    [12]
    J. Wang, L. Qiao, Y. Ye, and Y. Chen,“Fractional envelope analysis for rolling element bearing weak fault feature extraction,” IEEE/CAA Journal of Automatica Sinica, vol. 4, no. 2, pp. 1-8, 2016.
    [13]
    F. Jia, Y. Lei, J. Lin, X. Zhou, and N. Lu,“Deep neural networks: A promising tool for fault characteristic mining and intelligent diagnosis of rotating machinery with massive data,” Mechanical Systems and Signal Processing, vol. 72-73, pp. 303-315, 2016. doi:10.1016/j.ymssp.2015.10.025
    [14]
    C. Zhang, Z. Yang, X. He, and L. Deng,“Multimodal intelligence: Representation learning, information fusion, and applications,” IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 3, pp. 478-493, 2020. doi:10.1109/JSTSP.2020.2987728
    [15]
    G. Hu, H. Li, Y. Xia, and L. Luo,“A deep boltzmann machine and multi-grained scanning forest ensemble collaborative method and its application to industrial fault diagnosis,” Computers in Industry, vol. 100, pp. 287-296, 2018. doi:10.1016/j.compind.2018.04.002
    [16]
    S. Tang, S. Yuan, and Y. Zhu,“Deep learning-based intelligent fault diagnosis methods toward rotating machinery,” IEEE Access, vol. 8, pp. 9335-9346, 2020. doi:10.1109/ACCESS.2019.2963092
    [17]
    J. Tan, W. Lu, J. An, and X. Wan, “Fault diagnosis method study in roller bearing based on wavelet transform and stacked auto-encoder, ” in 27th Chinese Control and Decision Conference, Qingdao, China, May 2015, pp. 4608-4613.
    [18]
    Z. Liu, J. Wang, L. Duan, T. Shi, and Q. Fu, “Infrared image combined with cnn based fault diagnosis for rotating machinery, ” in 2017 International Conference on Sensing, Diagnostics, Prognostics, and Control, Shanghai, China, Aug. 2017, pp. 137-142.
    [19]
    Y. Dong, H. Wang, Y. Wang, and F. Chen, “Research on fault diagnosis algorithm of gas pressure regulator based on compressed sensing theory, ” in 29th Chinese Control And Decision Conference, Chongqing, China, May 2017, pp. 2702-2707.
    [20]
    Y. Lei, F. Jia, D. Kong, J. Lin, and S. Xing,“Opportunities and challenges of machinery intelligent fault diagnosis in big data era,” Journal of Mechanical Engineering, vol. 54, no. 5, pp. 94-104, 2018. doi:10.3901/JME.2018.05.094
    [21]
    M. H. W. Rosli and A. Cabrera,“Gestalt principles in multimodal data representation,” IEEE Computer Graphics & Applications, vol. 35, no. 2, pp. 80-87, 2015.
    [22]
    M. Ma, C. Sun, and X. Chen,“Deep coupling autoencoder for fault diagnosis with multimodal sensory data,” IEEE Transactions on Industrial Informatics, vol. 14, no. 3, pp. 1137-1145, 2018. doi:10.1109/TII.2018.2793246
    [23]
    R. Zhao, R. Yan, J. Wang, and K. Mao,“Learning to monitor machine health with convolutional bi-directional LSTM networks,” Sensors, vol. 17, no. 2, pp. 273, 2017. doi:10.3390/s17020273
    [24]
    S. Zhang, Y. Wang, M. Liu, and Z. Bao,“Data-based line trip fault prediction in power systems using LSTM networks and SVM,” IEEE Access, vol. 6, pp. 7675-7686, 2017.
    [25]
    N. Wang, F. Yang, R. Zhang, and F. Gao, “Intelligent fault diagnosis for chemical processes using deep learning multimodel fusion,” IEEE Transactions on Cybernetics, 2020. DOI: 10.1109/TCYB.2020.3038832.
    [26]
    Z. Li, L. Tian, Q. Jiang, and X. Yan,“Fault diagnostic method based on deep learning and multimodel feature fusion for complex industrial processes,” Industrial and Engineering Chemistry Research, vol. 59, no. 40, pp. 18061-18069, 2020. doi:10.1021/acs.iecr.0c03082
    [27]
    W. Xu, L. Jing, J. Tan, and L. Dou,“A multimodel decision fusion method based on DCNN-IDST for fault diagnosis of rolling bearing,” Shock and Vibration, vol. 2020, pp. 1-12, 2020.
    [28]
    D. Yu and J. Li,“Recent progresses in deep learning based acoustic models,” IEEE/CAA Journal of Automatica Sinica, vol. 4, no. 3, pp. 396-409, 2017. doi:10.1109/JAS.2017.7510508
    [29]
    Y. Xie and T. Zhang,“Fault diagnosis for rotating machinery based on convolutional neural network and empirical mode decomposition,” Shock and Vibration, vol. 2017, pp. 1-12, 2017.
    [30]
    B. Bayar and M. C. Stamm, “Design principles of convolutional neural networks for multimedia forensics, ” in Media Watermarking, Security, and Forensics, Burlingame, CA, USA, 2017, pp. 77-86.
    [31]
    F. Wang, H. Jiang, H. Shao, W. Duan, and S. Wu,“An adaptive deep convolutional neural network for rolling bearing fault diagnosis,” Measurement Science and Technology, vol. 28, no. 9, pp. 17-35, 2017.
    [32]
    W. Jia, M. Li, M. Zhu, and W. Jun,“License plate character recognition based on stacked denoising autoencoder,” Computer Engineering and Design, vol. 37, no. 3, pp. 751-766, 2016.
    [33]
    Z. Chen and Z. Li,“Fault diagnosis method of rotating machinery based on stacked denoising autoencoder,” Journal of Intelligent and Fuzzy Systems, vol. 34, no. 6, pp. 3443-3449, 2018. doi:10.3233/JIFS-169524
    [34]
    M. Abadi,“Tensorflow: learning functions at scale,” Association for Computing Machinery Special Interest Group on Programming Languages Notices, vol. 51, no. 9, pp. 1-11, 2016.
    [35]
    H. Friedman,“Greedy function approximation: A gradient boosting machine,” Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001. doi:10.1214/aos/1013203450
    [36]
    R. Ethan, R. Vincent, and K. Kurt, “ORB: An efficient alternative to SIFT or SURF, ” in 2011 International Conference on Computer Vision, Barcelona, Spain, Nov. 2011, pp. 2564-2571.
    • Relative Articles
  • Supplements (0)
  • Cited By
  • 加载中

Catalog

    通讯作者:陈斌, bchen63@163.com
    • 1.

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(10)/Tables(2)

    Get Citation
    PDF
    XML

    Article Metrics

    Article views (1326) PDF downloads(76) Cited by()
    Proportional views
    Related

    Address:5 Zhongguancun South Street, Haidian District, Beijing

    Tel:86-10-68914374

    Email:blgywb@bit.edu.cn

    Copy Right © Journal of Beijing Institute of Technology

    Supported by:Beijing Renhe Information Technology Co. Ltd

    /

    Return
    Return
      Baidu
      map