Deep Learning Models for Retinal Blood Vessels Segmentation: A Review <s> B. REGULARIZATION <s> When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This "overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random "dropout" gives big improvements on many benchmark tasks and sets new records for speech and object recognition. <s> BIB001 </s> Deep Learning Models for Retinal Blood Vessels Segmentation: A Review <s> B. REGULARIZATION <s> In the last few years, deep learning has led to very good performance on a variety of problems, such as visual recognition, speech recognition and natural language processing. Among different types of deep neural networks, convolutional neural networks have been most extensively studied. Due to the lack of training data and computing power in early days, it is hard to train a large high-capacity convolutional neural network without overfitting. After the rapid growth in the amount of the annotated data and the recent improvements in the strengths of graphics processor units (GPUs), the research on convolutional neural networks has been emerged swiftly and achieved state-of-the-art results on various tasks. In this paper, we provide a broad survey of the recent advances in convolutional neural networks. Besides, we also introduce some applications of convolutional neural networks in computer vision. <s> BIB002
One of the most problematic issues regarding CNN training is overfitting. Overfitting happens when the model fits too well to the training dataset, and it cannot generalize to new examples that were not in the training dataset. So, overfitting is an unneglectable problem in deep CNNs BIB002 . There are many proposed solutions to reduce the overfitting effectively, such as Dropout BIB001 , L1 regularization, and L2 regularization BIB002 . In deep learning, Dropout is widely used as regularization after the fully connected layers. It deletes or deactivates some neurons so that not all connections between the layers are activated at that time during training. It can also be applied after the convolutional layers. However, it is not preferable to add them in the first layers because dropout causes information to get lost. And if the information is lost in the first layers, it will be also lost for the whole network and this will affect the performance of the network. During testing time, dropout layers are bypassed and they are not active.