Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> I. INTRODUCTION <s> Abstract The capability to accurately predict the remaining life of a rolling element bearing is prerequisite to the optimal maintenance of rotating machinery performance in terms of cost and productivity. Due to the probabilistic nature of bearing integrity and operation condition, reliable estimation of a bearing's remaining life presents a challenging aspect in the area of maintenance optimisation and catastrophic failure avoidance. Previous study has developed an adaptive prognostic methodology to estimate the rate of bearing defect growth based on a deterministic defect-propagation model. However, deterministic models are inadequate in addressing the stochastic nature of defect-propagation. In this paper, a stochastic defect-propagation model is established by instituting a lognormal random variable in a deterministic defect-propagation rate model. The resulting stochastic model is calibrated on-line by a recursive least-squares (RLS) approach without the requirement of a priori knowledge on bearing characteristics. An augmented stochastic differential equation vector is developed with the consideration of model uncertainties, parameter estimation errors, and diagnostic model inaccuracies. It involves two ordinary differential equations for the first and second moments of its random variables. Solving the two equations gives the mean path of defect propagation and its dispersion at any instance. This approach is suitable for on-line monitoring, remaining life prediction, and decision making for optimal maintenance scheduling. The methodology has been verified by numerical simulations and the experimental testing of bearing fatigue life. <s> BIB001 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> I. INTRODUCTION <s> A physics-based approach for diagnostics and prognostics using integrated observers and life models is presented. Observers are filters based on physical models of machine- fault combinations and use measured machine signatures to identify and characterize the state of a machine. Observers are adaptively deployed as a machine wears and can be coupled with one another to handle interacting conditions and faults. The scheme is detailed using the fault of a cracked rotor shaft that interacts with gravity and imbalance. Observers for shaft cracking and imbalance are presented. The observers provide machine condition and fault strengths to life models used to determine remaining machine life. A life model based on the Forman crack growth law of linear elastic fracture mechanics is developed to determine the number of machine cycles remaining until catastrophic failure.© (2002) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only. <s> BIB002 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> I. INTRODUCTION <s> Condition-based maintenance (CBM) is a maintenance program that recommends maintenance decisions based on the information collected through condition monitoring. It consists of three main steps: data acquisition, data processing and maintenance decision-making. Diagnostics and prognostics are two important aspects of a CBM program. Research in the CBM area grows rapidly. Hundreds of papers in this area, including theory and practical applications, appear every year in academic journals, conference proceedings and technical reports. This paper attempts to summarise and review the recent research and developments in diagnostics and prognostics of mechanical systems implementing CBM with emphasis on models, algorithms and technologies for data processing and maintenance decision-making. Realising the increasing trend of using multiple sensors in condition monitoring, the authors also discuss different techniques for multiple sensor data fusion. The paper concludes with a brief discussion on current practices and possible future trends of CBM. <s> BIB003 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> I. INTRODUCTION <s> To achieve its impressive performance in tasks such as speech perception or object recognition, the brain extracts multiple levels of representation from the sensory input. Backpropagation was the first computationally efficient model of how neural networks could learn multiple layers of representation, but it required labeled training data and it did not work well in deep networks. The limitations of backpropagation learning can now be overcome by using multilayer neural networks that contain top-down connections and training them to generate sensory data rather than to classify it. Learning multilayer generative models might seem difficult, but a recent discovery makes it easy to learn nonlinear distributed representations one layer at a time. <s> BIB004 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> I. INTRODUCTION <s> We describe a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense (grammatically and semantically) using a language model. The entire network is trained jointly on all these tasks using weight-sharing, an instance of multitask learning. All the tasks use labeled data except the language model which is learnt from unlabeled text and represents a novel form of semi-supervised learning for the shared tasks. We show how both multitask learning and semi-supervised learning improve the generalization of the shared tasks, resulting in state-of-the-art-performance. <s> BIB005 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> I. INTRODUCTION <s> The promise of unsupervised learning methods lies in their potential to use vast amounts of unlabeled data to learn complex, highly nonlinear models with millions of free parameters. We consider two well-known unsupervised learning models, deep belief networks (DBNs) and sparse coding, that have recently been applied to a flurry of machine learning applications (Hinton & Salakhutdinov, 2006; Raina et al., 2007). Unfortunately, current learning algorithms for both models are too slow for large-scale applications, forcing researchers to focus on smaller-scale models, or to use fewer training examples. In this paper, we suggest massively parallel methods to help resolve these problems. We argue that modern graphics processors far surpass the computational capabilities of multicore CPUs, and have the potential to revolutionize the applicability of deep unsupervised learning methods. We develop general principles for massively parallelizing unsupervised learning tasks using graphics processors. We show that these principles can be applied to successfully scaling up learning algorithms for both DBNs and sparse coding. Our implementation of DBN learning is up to 70 times faster than a dual-core CPU implementation for large models. For example, we are able to reduce the time required to learn a four-layer DBN with 100 million free parameters from several weeks to around a single day. For sparse coding, we develop a simple, inherently parallel algorithm, that leads to a 5 to 15-fold speedup over previous methods. <s> BIB006 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> I. INTRODUCTION <s> Prognosis of hybrid systems is a challenging problem because multiple faults may happen simultaneously at a mode where these faults have different detectability. In other words, at the fault-initiating mode, some of the faults are detectable while others are nondetectable. As a result, decision making based on only one observation of abnormal behavior is not reliable under this condition. This paper focuses on the development of a model-based prognosis framework for hybrid systems where a dynamic fault isolation scheme is proposed to facilitate the prognostic tasks. The degradation behavior of each faulty component is mode dependent and can be estimated by a hybrid differential evolution algorithm. Thereafter, the remaining useful life of the faulty component that varies with different operating modes is calculated by using both the estimated degradation model and the user-selected failure threshold. Experiments are carried out to validate the key concepts of the developed methods, and results suggest the effectiveness. <s> BIB007 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> I. INTRODUCTION <s> Motivation: Alternative splicing (AS) is a regulated process that directs the generation of different transcripts from single genes. A computational model that can accurately predict splicing patterns based on genomic features and cellular context is highly desirable, both in understanding this widespread phenomenon, and in exploring the effects of genetic variations on AS. Methods: Using a deep neural network, we developed a model inferred from mouse RNA-Seq data that can predict splicing patterns in individual tissues and differences in splicing patterns across tissues. Our architecture uses hidden variables that jointly represent features in genomic sequences and tissue types when making predictions. A graphics processing unit was used to greatly reduce the training time of our models with millions of parameters. Results: We show that the deep architecture surpasses the performance of the previous Bayesian method for predicting AS patterns. With the proper optimization procedure and selection of hyperparameters, we demonstrate that deep architectures can be beneficial, even with a moderately sparse dataset. An analysis of what the model has learned in terms of the genomic features is presented. Contact: frey@psi.toronto.edu Supplementary information: Supplementary data are available at Bioinformatics online. <s> BIB008 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> I. INTRODUCTION <s> In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarizes relevant work, much of it from the previous millennium. Shallow and Deep Learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks. <s> BIB009 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> I. INTRODUCTION <s> This paper provides an overview of the recent developments in data-based techniques focused on modern industrial applications. As one of the hottest research topics for complicated processes, the data-based techniques have been rapidly developed over the past two decades and widely used in numerous industrial sectors nowadays. The core of data-based techniques is to take full advantage of the huge amounts of available process data, aiming to acquire the useful information within. Compared with the well-developed model-based approaches, data-based techniques provide efficient alternative solutions for different industrial issues under various operating conditions. The main objective of this paper is to review and summarize the recent achievements in data-based techniques, especially for complicated industrial applications, thus providing a referee for further study on the related topics both from academic and practical points of view. This paper begins with a brief evolutionary overview of data-based techniques in the last two decades. Then, the methodologies only based on process measurements and the model-data integrated techniques will be further introduced. The recent developments for modern industrial applications are, respectively, presented mainly from perspectives of monitoring and control. The new trends of data-based technique as well as potential application fields are finally discussed. <s> BIB010 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> I. INTRODUCTION <s> Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors. <s> BIB011 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> I. INTRODUCTION <s> This book develops the core system science needed to enable the development of a complex industrial internet of things/manufacturing cyber-physical systems (IIoT/M-CPS). Gathering contributions from leading experts in the field with years of experience in advancing manufacturing, it fosters a research community committed to advancing research and education in IIoT/M-CPS and to translating applicable science and technology into engineering practice. Presenting the current state of IIoT and the concept of cybermanufacturing, this book is at the nexus of research advances from the engineering and computer and information science domains. Readers will acquire the core system science needed to transform to cybermanufacturing that spans the full spectrum from ideation to physical realization. <s> BIB012 </s> Deep Learning and Its Applications to Machine Health Monitoring: A Survey <s> I. INTRODUCTION <s> State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet [1] and Fast R-CNN [2] have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with ’attention’ mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model [3] , our detection system has a frame rate of 5 fps ( including all steps ) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available. <s> BIB013
I NDUSTRIAL Internet of Things (IoT) and data-driven techniques have been revolutionizing manufacturing by enabling computer networks to gather the huge amount of data from connected machines and turn the big machinery data into actionable information BIB010 , BIB012 ,  . As a key component in modern manufacturing system, machine health monitoring has fully embraced the big data revolution. Compared to top-down modeling provided by the traditional physics-based models BIB001 , BIB002 , BIB007 , data-driven machine health monitoring systems offer a new paradigm of bottom-up solution for detection of faults after the occurrence of certain failures (diagnosis) and predictions of the future working conditions and the remaining useful life (prognosis) BIB010 , BIB003 . As we know, the complexity and noisy working condition hinder the construction of physical models. And most of these physicsbased models are unable to be updated with on-line measured data, which limits their effectiveness and flexibility. On the other hand, with significant development of sensors, sensor networks and computing systems, data-driven machine health monitoring models have become more and more attractive. To extract useful knowledge and make appropriate decisions from big data, machine learning techniques have been regarded as a powerful solution. As the hottest subfield of machine R. Yan is the corresponding author. E-mail: ruqiang@seu.edu.cn This manuscript has been submitted to IEEE Transactions on Neural Networks and Learning Systems learning, deep learning is able to act as a bridge connecting big machinery data and intelligent machine health monitoring. As a branch of machine learning, deep learning attempts to model high level representations behind data and classify(predict) patterns via stacking multiple layers of information processing modules in hierarchical architectures. Recently, deep learning has been successfully adopted in various areas such as computer vision, automatic speech recognition, natural language processing, audio recognition and bioinformatics BIB013 , BIB005 ,  , BIB008 . In fact, deep learning is not a new idea, which even dates back to the 1940s BIB009 , BIB011 . The popularity of deep learning today can be contributed to the following points: * Increasing Computing Power: the advent of graphics processor unit (GPU), the lowered cost of hardware, the better software infrastructure and the faster network connectivity all reduce the required running time of deep learning algorithms significantly. For example, as reported in BIB006 , the time required to learn a four-layer DBN with 100 million free parameters can be reduced from several weeks to around a single day. * Increasing Data Size: there is no doubt that the era of Big Data is coming. Our activities are almost all digitized, recorded by computers and sensors, connected to Internet, and stored in cloud. As pointed out in BIB010 that in industry-related applications such as industrial informatics and electronics, almost 1000 exabytes are generated per year and a 20-fold increase can be expected in the next ten years. The study in  predicts that 30 billion devices will be connected by 2020. Therefore, the huge amount of data is able to offset the complexity increase behind deep learning and improve its generalization capability. * Advanced Deep Learning Research: the first breakthrough of deep learning is the pre-training method in an unsupervised way BIB004 , where Hinton proposed to pre-train one layer at a time via restricted Boltzmann machine (RBM) and then fine-tune using backpropagation. This has been proven to be effective to train multilayer neural networks. Considering the capability of deep learning to address largescale data and learn high-level representation, deep learning can be a powerful and effective solution for machine health monitoring systems (MHMS). Conventional data-driven MHMS usually consists of the following key parts: handcrafted feature design, feature extraction/selection and model training. The right set of features are designed, and then provided to some shallow machine learning algorithms including