Visual-Based Defect Detection and Classification Approaches for Industrial Applications—A SURVEY <s> Deep Learning for Defect Detection <s> We recently discovered stimulus-specific interactions between cell assemblies in cat primary visual cortex that could constitute a global linking principle for feature associations in sensory and motor systems: stimulus-induced oscillatory activities (35-80 Hz) in remote cell assemblies of the same and of different visual cortex areas mutually synchronize, if common stimulus features drive the assemblies simultaneously. Based on our neurophysiological findings we simulated feature linking via synchronizations in networks of model neurons. The networks consisted of two one-dimensional layers of neurons, coupled in a forward direction via feeding connections and in lateral and backward directions via modulatory linking connections. The models' performance is demonstrated in examples of region linking with spatiotemporally varying inputs, where the rhythmic activities in response to an input, that initially are uncorrelated, become phase locked. We propose that synchronization is a general principle for the coding of associations in and among sensory systems and that at least two distinct types of synchronization do exist: stimulus-forced (event-locked) synchronizations support crude instantaneous associations and stimulus-induced (oscillatory) synchronizations support more complex iterative association processes. In order to bring neural linking mechanisms into correspondence with perceptual feature linking, we introduce the concept of the linking field (association field) of a local assembly of visual neurons. The linking field extends the concept of the invariant receptive field (RF) of single neurons to the flexible association of RFs in neural assemblies. <s> BIB001 </s> Visual-Based Defect Detection and Classification Approaches for Industrial Applications—A SURVEY <s> Deep Learning for Defect Detection <s> The pulse coupled neural network (PCNN) models are described. The linking field modulation term is shown to be a universal feature of any biologically grounded dendritic model. Applications and implementations of PCNN's are reviewed. Application based variations and simplifications are summarized. The PCNN image decomposition (factoring) model is described in new detail. <s> BIB002 </s> Visual-Based Defect Detection and Classification Approaches for Industrial Applications—A SURVEY <s> Deep Learning for Defect Detection <s> An automatic parameter setting method of a simplified pulse coupled neural network (SPCNN) is proposed here. Our method successfully determines all the adjustable parameters in SPCNN and does not need any training and trials as required by previous methods. In order to achieve this goal, we try to derive the general formulae of dynamic threshold and internal activity of the SPCNN according to the dynamic properties of neurons, and then deduce the sub-intensity range expression of each segment based on the general formulae. Besides, we extract information from an input image, such as the standard deviation and the optimal histogram threshold of the image, and attempt to build a direct relation between the dynamic properties of neurons and the static properties of each input image. Finally, the experimental segmentation results of the gray natural images from the Berkeley Segmentation Dataset, rather than synthetic images, prove the validity and efficiency of our proposed automatic parameter setting method of SPCNN. <s> BIB003 </s> Visual-Based Defect Detection and Classification Approaches for Industrial Applications—A SURVEY <s> Deep Learning for Defect Detection <s> Convolutional neural networks (CNNs) achieved impressive recognition rates in image classification tasks recently. In order to exploit those capabilities, we trained CNNs on a database of photometric stereo images of metal surface defects, i.e. rail defects. Those defects are cavities in the rail surface and are indication for further surface degradation right up to rail break. Due to security issues, defects have to be recognized early in order to take countermeasures in time. By means of differently colored light-sources illuminating the rail surfaces from different and constant directions, those cavities are made visible in a photometric dark-field setup. So far, a model-based approach has been used for image classification, which expressed the expected reflection properties of surface defects in contrast to non-defects. In this work, we experimented with classical CNNs trained in pure supervised manner and also explored the impact of regularization methods such as unsupervised layer-wise pre-training and training data-set augmentation. The classical CNN already distinctly outperforms the model-based approach. Moreover, regularization methods yet yield further improvements. <s> BIB004 </s> Visual-Based Defect Detection and Classification Approaches for Industrial Applications—A SURVEY <s> Deep Learning for Defect Detection <s> Abstract Fast and reliable industrial inspection is a main challenge in manufacturing scenarios. However, the defect detection performance is heavily dependent on manually defined features for defect representation. In this contribution, we investigate a new paradigm from machine learning, namely deep machine learning by examining design configurations of deep Convolutional Neural Networks (CNN) and the impact of different hyper-parameter settings towards the accuracy of defect detection results. In contrast to manually designed image processing solutions, deep CNN automatically generate powerful features by hierarchical learning strategies from massive amounts of training data with a minimum of human interaction or expert process knowledge. An application of the proposed method demonstrates excellent defect detection results with low false alarm rates. <s> BIB005 </s> Visual-Based Defect Detection and Classification Approaches for Industrial Applications—A SURVEY <s> Deep Learning for Defect Detection <s> Machine vision is an excellent tool for inspecting a variety of items such as textiles, fruit, printed circuit boards, electrical components, labels, integrated circuits, machine tools, etc. This paper presents an intelligent system that incorporates machine vision with artificial intelligent networks to automatically inspect thermal fuses. An effective inspection flow is proposed to detect four commonly seen defects, including black-dot, small-head, bur, and flake during the production of thermal fuses. Backpropagation neural networks and learning vector quantization performance is compared in detecting the bur defect because of its illegibility. Different numbers of defective samples were screened out from a production line in a case study company and used to demonstrate the efficacy of the proposed system. Currently, the proposed inspection system is operating at the case study company, replacing four to six human inspectors. The system not only ensures the quality of the thermal fuses produced, but also reduced the cost of manual visual inspection. <s> BIB006 </s> Visual-Based Defect Detection and Classification Approaches for Industrial Applications—A SURVEY <s> Deep Learning for Defect Detection <s> Abstract Automated pavement distress detection and classification has remained one of the high-priority research areas for transportation agencies. In this paper, we employed a Deep Convolutional Neural Network (DCNN) trained on the ‘big data’ ImageNet database, which contains millions of images, and transfer that learning to automatically detect cracks in Hot-Mix Asphalt (HMA) and Portland Cement Concrete (PCC) surfaced pavement images that also include a variety of non-crack anomalies and defects. Apart from the common sources of false positives encountered in vision based automated pavement crack detection, a significantly higher order of complexity was introduced in this study by trying to train a classifier on combined HMA-surfaced and PCC-surfaced images that have different surface characteristics. A single-layer neural network classifier (with ‘adam’ optimizer) trained on ImageNet pre-trained VGG-16 DCNN features yielded the best performance. <s> BIB007 </s> Visual-Based Defect Detection and Classification Approaches for Industrial Applications—A SURVEY <s> Deep Learning for Defect Detection <s> State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet [1] and Fast R-CNN [2] have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with ’attention’ mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model [3] , our detection system has a frame rate of 5 fps ( including all steps ) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available. <s> BIB008 </s> Visual-Based Defect Detection and Classification Approaches for Industrial Applications—A SURVEY <s> Deep Learning for Defect Detection <s> Abstract Smart manufacturing refers to using advanced data analytics to complement physical science for improving system performance and decision making. With the widespread deployment of sensors and Internet of Things, there is an increasing need of handling big manufacturing data characterized by high volume, high velocity, and high variety. Deep learning provides advanced analytics tools for processing and analysing big manufacturing data. This paper presents a comprehensive survey of commonly used deep learning algorithms and discusses their applications toward making manufacturing “smart”. The evolvement of deep learning technologies and their advantages over traditional machine learning are firstly discussed. Subsequently, computational methods based on deep learning are presented specially aim to improve system performance in manufacturing. Several representative deep learning models are comparably discussed. Finally, emerging topics of research on deep learning are highlighted, and future trends and challenges associated with deep learning for smart manufacturing are summarized. <s> BIB009 </s> Visual-Based Defect Detection and Classification Approaches for Industrial Applications—A SURVEY <s> Deep Learning for Defect Detection <s> Nowadays, there is a strong demand for inspection systems integrating both high sensitivity under various testing conditions and advanced processing allowing automatic identification of the examined object state and detection of threats. This paper presents the possibility of utilization of a magnetic multi-sensor matrix transducer for characterization of defected areas in steel elements and a deep learning based algorithm for integration of data and final identification of the object state. The transducer allows sensing of a magnetic vector in a single location in different directions. Thus, it enables detecting and characterizing any material changes that affect magnetic properties regardless of their orientation in reference to the scanning direction. To assess the general application capability of the system, steel elements with rectangular-shaped artificial defects were used. First, a database was constructed considering numerical and measurements results. A finite element method was used to run a simulation process and provide transducer signal patterns for different defect arrangements. Next, the algorithm integrating responses of the transducer collected in a single position was applied, and a convolutional neural network was used for implementation of the material state evaluation model. Then, validation of the obtained model was carried out. In this paper, the procedure for updating the evaluated local state, referring to the neighboring area results, is presented. Finally, the results and future perspective are discussed. <s> BIB010 </s> Visual-Based Defect Detection and Classification Approaches for Industrial Applications—A SURVEY <s> Deep Learning for Defect Detection <s> Defect inspection is a vital part of the production process to control the quality of LED chip. On the one hand, traditional methods are time-consuming, which rely on models badly and require rich operation experience. On the other hand, defect localization cannot be achieved by using traditional networks. To solve these problems, we achieve the application of convolutional neural network (CNN) for LED chip defect inspection. Built in the CNN, a class activation mapping technique is proposed to localize defect regions without using region-level human annotations. Further, LED chip datasets are collected for training the CNN. It is worth to emphasize that the chip defect classification and localization tasks are completed in a single CNN which is very fast and convenient. The proposed CNN based defect inspector named LEDNet achieves impressively high performance on the inspection of LED chip defects (line blemishes and scratch marks) with an inaccuracy of 5.04%, localizing exact defect regions as well. <s> BIB011 </s> Visual-Based Defect Detection and Classification Approaches for Industrial Applications—A SURVEY <s> Deep Learning for Defect Detection <s> Computer vision-based techniques were developed to overcome the limitations of visual inspection by trained human resources and to detect structural damage in images remotely, but most methods detect only specific types of damage, such as concrete or steel cracks. To provide quasi real-time simultaneous detection of multiple types of damages, a Faster Region-based Convolutional Neural Network (Faster R-CNN)-based structural visual inspection method is proposed. To realize this, a database including 2,366 images (with 500 × 375 pixels) labeled for five types of damages—concrete crack, steel corrosion with two levels (medium and high), bolt corrosion, and steel delamination—is developed. Then, the architecture of the Faster R-CNN is modified, trained, validated, and tested using this database. Results show 90.6%, 83.4%, 82.1%, 98.1%, and 84.7% average precision (AP) ratings for the five damage types, respectively, with a mean AP of 87.8%. The robustness of the trained Faster R-CNN is evaluated and demonstrated using 11 new 6,000 × 4,000-pixel images taken of different structures. Its performance is also compared to that of the traditional CNN-based method. Considering that the proposed method provides a remarkably fast test speed (0.03 seconds per image with 500 × 375 resolution), a framework for quasi real-time damage detection on video using the trained networks is developed. <s> BIB012 </s> Visual-Based Defect Detection and Classification Approaches for Industrial Applications—A SURVEY <s> Deep Learning for Defect Detection <s> Due to object detection’s close relationship with video analysis and image understanding, it has attracted much research attention in recent years. Traditional object detection methods are built on handcrafted features and shallow trainable architectures. Their performance easily stagnates by constructing complex ensembles that combine multiple low-level image features with high-level context from object detectors and scene classifiers. With the rapid development in deep learning, more powerful tools, which are able to learn semantic, high-level, deeper features, are introduced to address the problems existing in traditional architectures. These models behave differently in network architecture, training strategy, and optimization function. In this paper, we provide a review of deep learning-based object detection frameworks. Our review begins with a brief introduction on the history of deep learning and its representative tool, namely, the convolutional neural network. Then, we focus on typical generic object detection architectures along with some modifications and useful tricks to improve detection performance further. As distinct specific detection tasks exhibit different characteristics, we also briefly survey several specific tasks, including salient object detection, face detection, and pedestrian detection. Experimental analyses are also provided to compare various methods and draw some meaningful conclusions. Finally, several promising directions and tasks are provided to serve as guidelines for future work in both object detection and relevant neural network-based learning systems. <s> BIB013 </s> Visual-Based Defect Detection and Classification Approaches for Industrial Applications—A SURVEY <s> Deep Learning for Defect Detection <s> Automated surface-anomaly detection using machine learning has become an interesting and promising area of research, with a very high and direct impact on the application domain of visual inspection. Deep-learning methods have become the most suitable approaches for this task. They allow the inspection system to learn to detect the surface anomaly by simply showing it a number of exemplar images. This paper presents a segmentation-based deep-learning architecture that is designed for the detection and segmentation of surface anomalies and is demonstrated on a specific domain of surface-crack detection. The design of the architecture enables the model to be trained using a small number of samples, which is an important requirement for practical applications. The proposed model is compared with the related deep-learning methods, including the state-of-the-art commercial software, showing that the proposed approach outperforms the related methods on the specific domain of surface-crack detection. The large number of experiments also shed light on the required precision of the annotation, the number of required training samples and on the required computational cost. Experiments are performed on a newly created dataset based on a real-world quality control case and demonstrates that the proposed approach is able to learn on a small number of defected surfaces, using only approximately 25–30 defective training samples, instead of hundreds or thousands, which is usually the case in deep-learning applications. This makes the deep-learning method practical for use in industry where the number of available defective samples is limited. The dataset is also made publicly available to encourage the development and evaluation of new methods for surface-defect detection. <s> BIB014 </s> Visual-Based Defect Detection and Classification Approaches for Industrial Applications—A SURVEY <s> Deep Learning for Defect Detection <s> Abstract Automated surface inspection has become a hot topic with the rapid development of machine vision technologies. Traditional machine vision methods need experts to carefully craft image features for defect detection. This limits their applications to wider areas. The emerging convolutional neural networks (CNN) can automatically extract features and yield good results in many cases. However, the CNN-based image classification methods are more suitable for flat surface texture inspection. It is difficult to accurately locate small defects in geometrically complex products. Furthermore, the computational power required in CNN algorithms is usually high and it is not efficient to be implemented on embedded hardware. To solve these problems, a smart surface inspection system is proposed using faster R-CNN algorithm in the cloud-edge computing environment. The faster R-CNN as a CNN-based object detection method can efficiently identify defects in complex product images and the cloud-edge computing framework can provide fast computation speed and evolving algorithm models. A real industrial case study is presented to illustrate the effectiveness of the proposed method. The results show that the proposed method can provide high detection accuracy within a short time. <s> BIB015
Deep learning is one of fastest growing fields in computer sciences, due to its ability to solve highly complex problems BIB013 . The rich accumulation of traditional machine learning techniques resulted in the evolution of deep learning that also gained its inspiration from statistical learning. Most of the approaches mentioned in the previous sections are considered as traditional solutions, where the focus is on the explicitly engineered features which can be challenging to describe in complex cases. However, deep learning uses data representation learning to perform tasks, that transform data into complex, abstract representations that enable the features to be learnen for systems (e.g., feature learning). This ability of deep learning overcomes the requirement of complex features for a specific defect. Both deep learning and traditional machine learning are data-driven artificial intelligence techniques able to successfully model deterministic rules, which are often incomprehensible to humans and relationships between input and output. Moreover, deep learning disposes the capability of performing feature learning, model construction and model training, by selecting different kernels or tuning and optimizing parameters. In 2018, Wang et al. BIB009 summarized the capabilities of deep learning for smart manufacturing and highlighted how deep learning changes future trends in industry. Within the past years, a number of relevant studies have been published on defect detection solutions using deep learning BIB005 BIB010 BIB004 BIB014 BIB007 BIB011 . Lin, Hui et al. BIB011 developed a CNN called LEDNet to detect and classify defect on LED chips, where they achieved relatively low inaccuracy of 5.05%. Sun et al. BIB006 compared back-propagation neural networks and learning vector quantization performance in detecting the four commonly seen bur defect on thermal fuses. In 2015, Ren et al. BIB008 introduced a method by combining the region proposal network (RPN) and Faster Region-based Convolutional Neural Network (Faster R-CNN) for object detection to generate nearly cost-free region proposals. In BIB012 , the authors used a Faster R-CNN-based visual inspection method to detect and classify five defect types with 90.6%, 83.4%, 82.1%, 98.1%, and 84.7% average precisions. Notably, their method performed the task significantly more quickly than a traditional CNN based method, which is necessary for real-time implementation. Wang et al., BIB015 developed a faster R-CNN algorithm to solve the speed problem of CNNs and to locate small defects in geometrically complex products where they achieved 72% detection and 81% classification accuracy. Liu BIB002 . It was inspired by the study in BIB001 by Echorn about the synchronous dynamics of neuronal activity in cat visual cortex. Chen et al. BIB003 stated that, recently, PCNN models are the most potential method in image processing due to its high potential by resolving the problem of parameter estimation of segmentation problems. In 2018, Sacco et al.  developed a CNN based system for automatic quality control for fiber placement manufacturing, however they failed to achieve satisfactory results due to their small (only 200 samples/defect) training dataset that led them to over-fit their network. This factor is one of the most crucial factor by employing CNNs. To solve this common problem, Yang et al.  developed a promising and robust method as virtual defect rendering, that can solve the problem of small datasets. In a recent study  , Yang et al. developed a DCNN based system to detect and classify defects that can occur during laser welding in battery manufacturing. Besides that, they proposed a novel model called Visual Geometry Group (VGG) model to improve the efficiency of defect classification. Their test on 8000 samples with a 99.87% accuracy proved that the pre-trained VGG model has small model size, lower fault positive rate and shorter training time and prediction time. It is notable that their model is highly suitable for quality inspection in an industrial environment. Following the evolution of the industrial quality control field, there is an unequivocal need for general solutions to solve complex challenges that can be served by deep learning.