Deep Learning Models for Retinal Blood Vessels Segmentation: A Review <s> 2) ACTIVATION FUNCTION <s> Restricted Boltzmann machines were developed using binary stochastic hidden units. These can be generalized by replacing each binary unit by an infinite number of copies that all have the same weights but have progressively more negative biases. The learning and inference rules for these "Stepped Sigmoid Units" are unchanged. They can be approximated efficiently by noisy, rectified linear units. Compared with binary units, these units learn features that are better for object recognition on the NORB dataset and face verification on the Labeled Faces in the Wild dataset. Unlike binary units, rectified linear units preserve information about relative intensities as information travels through multiple layers of feature detectors. <s> BIB001 </s> Deep Learning Models for Retinal Blood Vessels Segmentation: A Review <s> 2) ACTIVATION FUNCTION <s> We introduce the"exponential linear unit"(ELU) which speeds up learning in deep neural networks and leads to higher classification accuracies. Like rectified linear units (ReLUs), leaky ReLUs (LReLUs) and parametrized ReLUs (PReLUs), ELUs alleviate the vanishing gradient problem via the identity for positive values. However, ELUs have improved learning characteristics compared to the units with other activation functions. In contrast to ReLUs, ELUs have negative values which allows them to push mean unit activations closer to zero like batch normalization but with lower computational complexity. Mean shifts toward zero speed up learning by bringing the normal gradient closer to the unit natural gradient because of a reduced bias shift effect. While LReLUs and PReLUs have negative values, too, they do not ensure a noise-robust deactivation state. ELUs saturate to a negative value with smaller inputs and thereby decrease the forward propagated variation and information. Therefore, ELUs code the degree of presence of particular phenomena in the input, while they do not quantitatively model the degree of their absence. In experiments, ELUs lead not only to faster learning, but also to significantly better generalization performance than ReLUs and LReLUs on networks with more than 5 layers. On CIFAR-100 ELUs networks significantly outperform ReLU networks with batch normalization while batch normalization does not improve ELU networks. ELU networks are among the top 10 reported CIFAR-10 results and yield the best published result on CIFAR-100, without resorting to multi-view evaluation or model averaging. On ImageNet, ELU networks considerably speed up learning compared to a ReLU network with the same architecture, obtaining less than 10% classification error for a single crop, single model network. <s> BIB002
CNNs have some linear components and nonlinear components. The activation functions are the nonlinear com- ponents which follow the convolutional layers to introduce the nonlinearities to the CNN to detect the nonlinear features and to improve the CNN performance. Rectified Linear Unit (ReLU) is one of the most popular activation function used in CNNs. It has been shown that CNNs can be trained efficiently using ReLU BIB001 . ReLU is defined as: where z is the input to the activation function and a is the output. ReLU keeps the positive part of the input and prunes the negative part to zero. Another version of ReLU is leaky ReLU (LReLU)  that defines a parameter λ in range (0, 1) to compress the negative part rather than mapping it to zero. Mathematically, LReLU is defined as: This makes a small and non-zero gradient when the unit is not active (negative value). Exponential Linear Unit (ELU) is another activation function that enables faster learning of the CNN and improves the accuracy of the classification task. Like ReLU and LReLU, ELU BIB002 sets the positive values to identity and the negative part is used for fast learning and it is robust to noise. Mathematically, ELU is defined as: where λ is a controlling parameter to saturate ELU for negative inputs. The last activation function we want to talk about is the sigmoid function (σ )  . Sigmoid function is used often in artificial neural networks to introduce the nonlinearity in the model. It takes real numbers and squashes them into range (0, 1). Mathematically, the sigmoid function is defined as: In general, the activation functions are applied to the output of convolutional layer in the CNN to add the nonlinearity to the output to project the values from some range to a desired range. The ''Activation Function'' term is biologically inspired. In real brain, neurons get signals from other neurons, and decided whether or not to fire by taking the cumulative input into account. This decision based on the cumulative input is the output from the activation function. Figure 8 shows the activation function we have discussed.