8 A Review of Major Application Areas of Differential Evolution <s> Training Integer Weight Neural Networks with Threshold Activations <s> Evolutionary neural network training algorithms are presented. These algorithms are applied to train neural networks with weight values confined to a narrow band of integers. We constrain the weights and biases in the range [-2/sup k+1/+1, 2/sup k-1/-1], for k=3, 4, 5, thus they can be represented by just k bits. Such neural networks are better suited for hardware implementation than the real weight ones. Mathematical operations that are easy to implement in software might often be very burdensome in the hardware and therefore more costly. Hardware-friendly algorithms are essential to ensure the functionality and cost effectiveness of the hardware implementation. To this end, in addition to the integer weights, the trained neural networks use threshold activation functions only, so hardware implementation is even easier. These algorithms have been designed keeping in mind that the resulting integer weights require less bits to be stored and the digital arithmetic operations between them are easier to be implemented in hardware. Obviously, if the network is trained in a constrained weight space, smaller weights are found and less memory is required. On the other hand, as we have found here, the network training procedure can be more effective and efficient when larger weights are allowed. Thus, for a given application a trade off between effectiveness and memory consumption has to be considered. Our intention is to present results of evolutionary algorithms on this difficult task. Based on the application of the proposed class of methods on classical neural network benchmarks, our experience is that these methods are effective and reliable. <s> BIB001 </s> 8 A Review of Major Application Areas of Differential Evolution <s> Training Integer Weight Neural Networks with Threshold Activations <s> In this paper, Parallel Evolutionary Algorithms for integer weight neural network training are presented. To this end, each processor is assigned a subpopulation of potential solutions. The subpopulations are independently evolved in parallel and occasional migration is employed to allow cooperation between them. The proposed algorithms are applied to train neural networks using threshold activation functions and weight values confined to a narrow band of integers. We constrain the weights and biases in the range [−3, 3], thus they can be represented by just 3 bits. Such neural networks are better suited for hardware implementation than the real weight ones. These algorithms have been designed keeping in mind that the resulting integer weights require less bits to be stored and the digital arithmetic operations between them are easier to be implemented in hardware. Another advantage of the proposed evolutionary strategies is that they are capable of continuing the training process ``on-chip'', if needed. Our intention is to present results of parallel evolutionary algorithms on this difficult task. Based on the application of the proposed class of methods on classical neural network problems, our experience is that these methods are effective and reliable. <s> BIB002
FNNs can be simulated in software, but to be utilized in real life applications, where high speed of execution is required, hardware implementation is needed. The natural implementation of an FNN -because of its modularity -is a parallel one. Hardwarefriendly algorithms are essential to ensure the functionality and cost effectiveness of the hardware implementation. Moreover, the need for hardware-friendly algorithms, which have the ability to cope with time-varying problems and real-time timing constraints, has been recently increased BIB002 . FNNs having integer weights and biases are easier and less expensive to implement in electronics as well as in optics and the storage of the integer weights is much easier to be achieved. Additionally, the use of threshold activation functions for all the hidden and output neurons, greatly reduces the complexity of the hardware implementation, because there is no need to design and implement complicated non-linear activation functions. Another advantage of the FNNs with integer weights and threshold activation functions is that the trained neural network is to some extend immune to noise in the training data. Such networks only capture the main feature of the training data. Low amplitude noise that possibly contaminates the training set cannot perturb the discrete weights, because those networks require relatively large variations to "jump" from one integer weight value to another. Mathematical operations that are easy to implement in software might often be very burdensome in the hardware and therefore more costly. This property reduces the amount of memory required for weight storage in digital electronic implementations. Additionally, it simplifies the digital multiplication operation. Finally, if inputs are restricted to the set {−1, 1} (bipolar inputs), the neurons in the first hidden layer require only sign changes during multiplication operations, and only integer additions. To apply DE to neural network training with integer weights, we initialize the individuals with N -dimensional integer weight vectors, following a uniform probability distribution, and evolve them over time. The only problem is that the mutation operator results in real weight vectors. As our aim is to maintain an integer weight population at each generation, each component of the mutant weight vector is rounded to the nearest integer. It must be noted that the evolutionary class of algorithms does not need the activation function to be differentiable and is suitable for training with threshold units BIB001 . In the first phase of our approach, the DE algorithms are used to train a neural network "offline", using sigmoid activation functions, such as: where λ is the gain parameter. This seems to be a good practice since the network is trained much faster with sigmoid functions. In the second phase we alter the gain of the sigmoid function in such a way that allows a mapping to a threshold unit network. Specifically, when the inputs are correctly classified and the network error is relatively small, the value of λ is increased in the sequence (1, 10, 20, 30, 40, 50, ∞) . Additional training might be necessary after each increase of λ. That justifies the additional iterations needed to train an FNN, using only threshold activation functions. This procedure is analogous to taking the limit of the sigmoid function as the gain parameter λ goes to infinity. Finally, the trained network uses only threshold activation functions and thus the complexity of the hardware implementation is greatly reduced. If new input data are introduced, training can be continued sequentially or in parallel "on-chip".