Deep Learning Models for Retinal Blood Vessels Segmentation: A Review <s> C. OPTIMIZATION AND LOSS FUNCTIONS <s> Support vector machines (SVMs) were originally designed for binary classification. How to effectively extend it for multiclass classification is still an ongoing research issue. Several methods have been proposed where typically we construct a multiclass classifier by combining several binary classifiers. Some authors also proposed methods that consider all classes at once. As it is computationally more expensive to solve multiclass problems, comparisons of these methods using large-scale problems have not been seriously conducted. Especially for methods solving multiclass SVM in one step, a much larger optimization problem is required so up to now experiments are limited to small data sets. In this paper we give decomposition implementations for two such "all-together" methods. We then compare their performance with three methods based on binary classifications: "one-against-all," "one-against-one," and directed acyclic graph SVM (DAGSVM). Our experiments indicate that the "one-against-one" and DAG methods are more suitable for practical use than the other methods. Results also show that for large problems methods by considering all data at once in general need fewer support vectors. <s> BIB001 </s> Deep Learning Models for Retinal Blood Vessels Segmentation: A Review <s> C. OPTIMIZATION AND LOSS FUNCTIONS <s> Training datasets for object detection problems are typically very large and Support Vector Machine (SVM) implementations are computationally complex. As opposed to these complex techniques, we use Stochastic Gradient Descent (SGD) algorithms that use only a single new training sample in each iteration and process samples in a stream-like fashion. We have incorporated SGD optimization in an object detection framework. The object detection problem is typically highly asymmetric, because of the limited variation in object appearance, compared to the background. Incorporating SGD speeds up the optimization process significantly, requiring only a single iteration over the training set to obtain results comparable to state-of-the-art SVM techniques. SGD optimization is linearly scalable in time and the obtained speedup in computation time is two to three orders of magnitude. We show that by considering only part of the total training set, SGD converges quickly to the overall optimum. <s> BIB002 </s> Deep Learning Models for Retinal Blood Vessels Segmentation: A Review <s> C. OPTIMIZATION AND LOSS FUNCTIONS <s> Deep and recurrent neural networks (DNNs and RNNs respectively) are powerful models that were considered to be almost impossible to train using stochastic gradient descent with momentum. In this paper, we show that when stochastic gradient descent with momentum uses a well-designed random initialization and a particular type of slowly increasing schedule for the momentum parameter, it can train both DNNs and RNNs (on datasets with long-term dependencies) to levels of performance that were previously achievable only with Hessian-Free optimization. We find that both the initialization and the momentum are crucial since poorly initialized networks cannot be trained with momentum and well-initialized networks perform markedly worse when the momentum is absent or poorly tuned. ::: ::: Our success training these models suggests that previous attempts to train deep and recurrent neural networks from random initializations have likely failed due to poor initialization schemes. Furthermore, carefully tuned momentum methods suffice for dealing with the curvature issues in deep and recurrent network training objectives without the need for sophisticated second-order methods. <s> BIB003 </s> Deep Learning Models for Retinal Blood Vessels Segmentation: A Review <s> C. OPTIMIZATION AND LOSS FUNCTIONS <s> Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters. <s> BIB004 </s> Deep Learning Models for Retinal Blood Vessels Segmentation: A Review <s> C. OPTIMIZATION AND LOSS FUNCTIONS <s> Cross-entropy loss together with softmax is arguably one of the most common used supervision components in convolutional neural networks (CNNs). Despite its simplicity, popularity and excellent performance, the component does not explicitly encourage discriminative learning of features. In this paper, we propose a generalized large-margin softmax (L-Softmax) loss which explicitly encourages intra-class compactness and inter-class separability between learned features. Moreover, L-Softmax not only can adjust the desired margin but also can avoid overfitting. We also show that the L-Softmax loss can be optimized by typical stochastic gradient descent. Extensive experiments on four benchmark datasets demonstrate that the deeply-learned features with L-softmax loss become more discriminative, hence significantly boosting the performance on a variety of visual classification and verification tasks. <s> BIB005 </s> Deep Learning Models for Retinal Blood Vessels Segmentation: A Review <s> C. OPTIMIZATION AND LOSS FUNCTIONS <s> Convolutional Neural Network (CNN) has been used successfully in solving different computer vision tasks such as classification, detection, and segmentation. This paper addresses the problem of estimating object depth from a single RGB image. While stereo depth estimation is a straightforward task, predicting depth map of an object from a single RGB image is a more challenging task due to the lack of information from various image cues. Prior work focuses on exploiting geometric information or hand-crafted features. Also, they proposed the L2 norm to perform the optimization of a CNN during the training process. Using the L2 norm in regression tasks for optimization will bias the model. To address this problem, we propose a regression model with a fully convolutional neural network. To achieve robustness to outliers, we optimize the model using Tukey's biweight loss function, which is an M-estimator that is robust against outliers. The predictions are given by a single fully CNN without any post-processing techniques. In our experiments, we show that the quantitative and the qualitative results of using Tukey's biweight loss for optimization are better than of using L2 norm. <s> BIB006 </s> Deep Learning Models for Retinal Blood Vessels Segmentation: A Review <s> C. OPTIMIZATION AND LOSS FUNCTIONS <s> Deep neural networks are currently among the most commonly used classifiers. Despite easily achieving very good performance, one of the best selling points of these models is their modular design - one can conveniently adapt their architecture to specific needs, change connectivity patterns, attach specialised layers, experiment with a large amount of activation functions, normalisation schemes and many others. While one can find impressively wide spread of various configurations of almost every aspect of the deep nets, one element is, in authors' opinion, underrepresented - while solving classification problems, vast majority of papers and applications simply use log loss. In this paper we try to investigate how particular choices of loss functions affect deep models and their learning dynamics, as well as resulting classifiers robustness to various effects. We perform experiments on classical datasets, as well as provide some additional, theoretical insights into the problem. In particular we show that L1 and L2 losses are, quite surprisingly, justified classification objectives for deep nets, by providing probabilistic interpretation in terms of expected misclassification. We also introduce two losses which are not typically used as deep nets objectives and show that they are viable alternatives to the existing ones. <s> BIB007
Training a CNN is a problem of global optimization. To find the best values of the weights for each layer, a loss function should be selected and minimized. To optimize the CNN parameters, Stochastic Gradient Descent (SGD) BIB002 is commonly used. To optimize a deep CNN, many steps should be done such as preparing the training dataset, designing the CNN and initializing the weights, choosing the loss function, and training the CNN using backpropagation to update the network weights. Below, we discuss each step briefly. It is well known that deep CNN almost depends on the availability of the training data. The more training data we have, the better trained model we can get. However, in some applications such as medical problems, the available datasets are small and we cannot rely on them to train a deep CNN. To overcome this issue, data augmentation [42] is applied to the training dataset. The purpose of the data augmentation is to increases the training data by transforming the images into new data without altering the nature of the content. The commonly used data augmentation methods include simple geometric transformations such as mirroring, shifting, scaling, rotating, and spatial cropping  . This method not only helps to overcome the problem of training data scarcity but also trains the network and updates the network weights more accurately. After preparing the training data, global data normalization is usually applied on the training dataset to transform the data to zero-mean and unit variance. However, during training and as the data flow deeper through the network, the distribution of the input data changes, which affects the training process and the network accuracy. Batch Normalization (BN) is applied on the input data in some layers to avoid this problem BIB004 . It fixes the means and the variances of the layer inputs by computing them after each mini-batch rather than the entire training set. Deep CNN has numerous of parameters to be optimized. A proper initializing of these parameters is important for fast convergence and to avoid the vanishing gradient problem. Many methods have been proposed to initialize the weights. In  , Krizhevsky et al. initialize the weights of the network from a zero-mean Gaussian distribution with standard deviation 0.01. Another weights initialization method is Xavier  . The idea in Xavier method is to initialize the weights from a Gaussian distribution with zero mean and a variance of 2 (n in +n out ) , where n in is the number of neurons feeding into the layer and n out is the number of the neurons that the result is fed to. Xavier method and its improved version allow deep networks to be trained and they converge fast. To optimize the CNN, backpropagation algorithm is used to train the CNN which uses gradient descent to update the CNN parameters. Among the optimization methods, Stochastic Gradient Descent (SGD) is commonly used to estimate the gradient on the basis of a single randomly picked example (x i , y j ) from the training dataset: where θ is the network parameters, x i is the input and y j is the output. L is the objective function that is used to train the network, and η is the learning rate. In practice, the network parameters are updated with respect to mini-batch. To guarantee the convergence and speedup the learning process, momentum  is proposed to make the current gradient depends on historical batches. The classical momentum update accumulates a velocity vector in the relevant direction, and it is given by: where ν i+1 is the current velocity vector and γ is the momentum term which is usually set to 0.9. Nesterov momentum BIB003 is another way of using momentum in gradient descent optimization that moves in the direction of the previous accumulated gradient, calculates the gradient and then updates the parameters. To update the network weights and set them to accurate values, we have to compare the predicted values with the ground-truth. To do this, we have to define a loss function that measures how close the prediction from the ground-truth is. Choosing the loss function is an important step because it is used to guide the training process of the CNN and measure the error between the predicted values and the ground-truth to correct and update the network weights. Also, it reflects the nature of the problem that the CNN has to solve (either a classification problem or a regression problem). There are many loss functions that are commonly used for the classification problems. Multiclass Support Vector Machine (SVM) BIB001 is set up so that the SVM wants the correct class of the input image to have the highest score within the incorrect classes' scores by some fixed margin . Suppose that the input image is x i and its label is y i and the output from the network activations is s = f (x i ; W ) . Consider the score of the j-th class is the j-th elements, s j = f (x i ; W ) j , then the Multiclass SVM for the i-th image is given by: Another commonly used loss function for classification is Softmax loss BIB005 . It is a combination of multinomial logistic loss and softmax. Consider that x i is the input and y i is the ground-truth (the class, k = 1, . . . , K ), the output from the activations is s = f (x i ; W ), and the prediction given from the softmax is: . In regression problems, it is important to predict real-valued quantities such as depth values in predicting depth images. L2 norm BIB007 is used commonly in regression problem, that computes the loss between the predicted values the ground-truth and then measures the L2 squared norm. It is defined as: where f is the output from the network activations y i is the corresponding ground-truth. Usually L2 is squared to make the gradient simpler and not to change the optimal parameters. L1 norm BIB007 is also used as a loss function in regression problem, and it is the sum of absolute values along each dimension. It is given by: It is important to note that choosing the loss functions affects the training process and the performance of the CNN. For example, it is known that L2 loss is less robust to the outliers because the outliers can introduce huge gradients. However, Tukey's biweight loss BIB006 is more robust against the outliers in regression problem.