A Survey of Outlier Detection Methodologies <s> Supervised neural methods <s> INTRODUCTION Humans and computers The structure of the brain Learning in machines The differences Summary PATTERN RECOGNITION Introduction Pattern recognition in perspective Pattern recognition-a definition Feature vectors and feature space Discriminant functions Classification techniques Linear classifiers Statistical techniques Pattern recognition-a summary THE BASIC NEURON Introduction Modeling the single neuron Learning in simple neurons The perceptron: a vectorial perspective The perceptron learning rule: proof Limitations of perceptrons The end of the line? Summary THE MULTILAYER PERCEPTRON Introduction Altering the perceptron model The new model The new learning rule The multilayer perceptron algorithm The XOR problem revisited Visualizing network behavior Multilayer perceptrons as classifiers Generalization Fault tolerance Learning difficulties Radial basis functions Applications Summary KOHONEN SELF-ORGANIZING NETWORKS Introduction The Kohonen algorithm Weight training Neighborhoods Reducing the neighborhood Learning vector quantization (LVQ) The phonetic typewriter Summary HOPFIELD NETWORKS Introduction The Hopfield model The energy landscape The Boltzmann machine Constraint satisfaction Summary ADAPTIVE RESONANCE THEORY Introduction Adaptive resonance theory (ART) Architecture and operation ART algorithm Training the ART network Classification Conclusion Summary of ART ASSOCIATIVE MEMORY Standard computer memory Implementing associative memory Implementation in RAMs RAMs and N-tupling Willshaw's associative net The ADAM system Kanerva's sparse distributed memory Bidirectional associative memories Conclusion Summary INTO THE LOOKING GLASS Overview Hardware and software implementations Optical computing Optical computing and neural networks INDEX <s> BIB001 </s> A Survey of Outlier Detection Methodologies <s> Supervised neural methods <s> The detection of novel or abnormal input vectors is of importance in many monitoring tasks, such as fault detection in complex systems and detection of abnormal patterns in medical diagnostics. We have developed a robust method for novelty detection, which aims to minimize the number of heuristically chosen thresholds in the novelty decision process. We achieve this by growing a gaussian mixture model to form a representation of a training set of "normal" system states. When previously unseen data are to be screened for novelty we use the same threshold as was used during training to define a novelty decision boundary. We show on a sample problem of medical signal processing that this method is capable of providing robust novelty decision boundaries and apply the technique to the detection of epileptic seizures within a data record. <s> BIB002 </s> A Survey of Outlier Detection Methodologies <s> Supervised neural methods <s> Novelty Detection techniques are concept-learning methods that proceed by recognizing positive instances of a concept rather than differentiating between its positive and negative instances. Novelty Detection approaches consequently require very few, if any, negative training instances. This paper presents a particular Novelty Detection approach to classification that uses a Redundancy Compression and Non-Redundancy Differentiation technique based on the [Gluck & Myers, 1993] model of the hippocampus, a part of the brain critically involved in learning and memory. In particular, this approach consists of training an autoencoder to reconstruct positive input instances at the output layer and then using this autoencoder to recognize novel instances. Classification is possible, after training, because positive instances are expected to be reconstructed accurately while negative instances are not. The purpose of this paper is to compare HIPPO, the system that implements this technique, to C4.5 and feedforward neural network classification on several applications. <s> BIB003 </s> A Survey of Outlier Detection Methodologies <s> Supervised neural methods <s> Neural networks are ideally suited for solving detection and classification problems. In the real world however often the number and labels of classes may be unknown. This is particularly true for the detection and classification of collected electromagnetic (EM) signals when new transmitter types are first turned on. Approaches that determine when new, never-seen-before, novel events are present in the system need to be considered. Described here is the development and application of a class-dependent elliptical basis function (CD-EBF) neural net developed to solve novelty detection problems. The neural net uses parameters derived from a linear model fit to the data as input. Application of the system to passive detection and classification of EM signals is shown. <s> BIB004
Supervised neural networks use the classification of the data to drive the learning process. The neural network uses the class to adjust the weights and thresholds to ensure the network can correctly classify the input. The input data is effectively modelled by the whole network with each point distributed across all nodes and the output representing the classification. For example, the data in Figure 4 is represented by the weights and connections of the entire network. Some supervised neural networks such as the multi-layer perceptron interpolate well but perform poorly for extrapolation so cannot classify unseen instances outside the bounds of the training set.  and  both exploit this for identifying novelties. Nairac identifies novelties in time-series data for fault diagnosis in vibration signatures of aircraft engines and Bishop monitors processes such as oil pipeline flows. The MLP is a feed forward network. Bishop and Nairac use a MLP with a single hidden layer. This allows the network to detect arbitrarily complex class boundaries. The nodes in the first layer define convex hulls analogous to the convex hull statistical methods discussed in the previous section. These then form the inputs for the second layer units which combine hulls to form complex classes. Bishop notes that the non-linearity offered by the MLP classifier provides a performance improvement compared to a linear technique. It is trained by minimising the square error between the actual value and the MLP output value given by equation (11): OUTLIER DETECTION METHODOLOGIES from  where t j is the target class, y j is the actual class, pðxÞ is the unconditional probability density which may be estimated by, for example, Parzen windows  ) (discussed in section 5) and y j ðx; wÞ is the function mapping. Provided the function mapping is flexible, if the network has sufficient hidden units, then the minimum occurs when y j ðx; wÞ ¼ pht j jxi. The outputs of the network are the regression of the target data conditioned with the input vector. The aim is thus to approximate the regression by minimising the sum-of-squares error using a finite training set. The approximation is highest where the density of pðxÞ is highest as this is where the error function penalises the network mapping if it differs from the regression. Where the density is low there is little penalisation. Bishop assumes that the MLP 'knows' the full scope of normality after training so the density pðxÞ is high, the network is interpolating and confidence is high (where confidence is given by equation (12): If a new input lies outside the trained distribution where the density pðxÞ is low and the network is extrapolating, the MLP will not recognise it and the confidence is low. Nairac uses the MLP to predict the expected next value based on the previous n values. This assumes that the next value is dependent only on the previous n values and ignores extraneous factors. If the actual value differs markedly from the predicted value then the system alerts the human monitoring it. BIB003 use an auto-associative neural network for type 3 novelty recognition. The auto associator network is also a feedforward perceptron-based network which uses supervised learning. Auto associators decrease the number of hidden nodes during network training to induce a bottleneck. The bottleneck reduces redundancies by focusing on the key attributes while maintaining and separating the essential input information and is analogous to principal component analysis. Japkowicz trains the auto-associator with normal data only. After training, the output nodes recreate the new exemplars applied to the network as inputs. The network will successfully recreate normal data but will generate a high re-creation error for novel data. Japkowicz counts the number of outputs different from the input. If this value exceeds a pre-specified threshold then the input is recognised as novel. Japkowicz sets the threshold at a level that minimises both false positive and false negative classifications.  demonstrated that auto-associative neural networks were more accurate than SOMs or Parzen windows for novelty detection. However, auto-associators suffer slow training as with the MLP and have various dataspecific parameters which must be set through empirical testing and refinement. Hopfield networks are also auto-associative and perform supervised learning. However, all node weights are þ1 or À1 and it is fully connected with no bottleneck. Hopfield nets are computationally efficient and their performance is evident when a large number of patterns is stored and the input vectors are high dimensional.  use a Hopfield network for type 3 novelty detection in mobile robots. The authors train the Hopfield network with pre-classified normal exemplars. The input pattern is applied to all nodes simultaneously unlike the MLP where the input ripples through the layers of the network. The output is fed back into the network as input to all nodes. Retrieval from a Hopfield network requires several network cycles while node weights update and to allow the network to settle and converge towards the 'best compromise solution' BIB001 . However, the energy calculation requires only the calculation of the sum of the weights see equation (13): where N is the number of neurons and thus has a fixed execution time regardless of the number of input patterns stored providing a computationally efficient retrieval mechanism. If the energy is above a threshold value calculated from the distribution model and the number of patterns stored then Crook and Hayes classify the input as novel. During empirical testing, the Hopfield network trained faster than Marsland's HSOM but the HSOM discerned finer-grained image details than the Hopfield network. The MLP uses hyper-planes to classify the data. In contrast, the supervised radial basis function (RBF) neural network in  and BIB004 uses hyper-ellipsoids defined by equation (14) for k dimensional inputs: where w jk is the weight from the kernel to the output, / is a Gaussian function (see equation (8)) and k Á Á Á k the Euclidean distance. RBF is a linear combination of basis functions similar to a Gaussian mixture model and guaranteed to produce a solution as hyper-ellipsoids guarantee linear separability  . Training the network is much faster than an MLP and requires two stages: the first stage clusters the input data into the hidden layer nodes using vector quantisation. The RBF derives the radius of the kernel from the data as with the Gaussian mixture model approach (see equation (8)) using the kernel centre and variance. The second training stage weights the hidden node outputs using least mean squares weighting to produce the required output classification from the network. BIB004 revise the network topology and vector quantisation so there is a group of hidden nodes exclusively assigned for each class with a separate vector quantisation metric for each class. This means that the groups of hidden nodes represent distinct categories unlike a conventional RBF where weight combinations of all hidden nodes represent the categories. The output from each group of nodes corresponds to the probability that the input vector belongs to the class represented by the group. Brotherton uses the adapted RBF for type 3 novelty recognition in electromagnetic data or machine vibration analysis. Brotherton's adaptation creates an incremental RBF. New nodes can be added to represent new classes just as new mixtures are added to BIB002 adaptive mixture model or new category nodes are added to evolutionary neural networks.