Deep Learning Models for Retinal Blood Vessels Segmentation: A Review <s> 1) CONVOLUTIONAL LAYER <s> We propose a novel deep network structure called "Network In Network" (NIN) to enhance model discriminability for local patches within the receptive field. The conventional convolutional layer uses linear filters followed by a nonlinear activation function to scan the input. Instead, we build micro neural networks with more complex structures to abstract the data within the receptive field. We instantiate the micro neural network with a multilayer perceptron, which is a potent function approximator. The feature maps are obtained by sliding the micro networks over the input in a similar manner as CNN; they are then fed into the next layer. Deep NIN can be implemented by stacking mutiple of the above described structure. With enhanced local modeling via the micro network, we are able to utilize global average pooling over feature maps in the classification layer, which is easier to interpret and less prone to overfitting than traditional fully connected layers. We demonstrated the state-of-the-art classification performances with NIN on CIFAR-10 and CIFAR-100, and reasonable performances on SVHN and MNIST datasets. <s> BIB001 </s> Deep Learning Models for Retinal Blood Vessels Segmentation: A Review <s> 1) CONVOLUTIONAL LAYER <s> Large Convolutional Network models have recently demonstrated impressive classification performance on the ImageNet benchmark Krizhevsky et al. [18]. However there is no clear understanding of why they perform so well, or how they might be improved. In this paper we explore both issues. We introduce a novel visualization technique that gives insight into the function of intermediate feature layers and the operation of the classifier. Used in a diagnostic role, these visualizations allow us to find model architectures that outperform Krizhevsky et al on the ImageNet classification benchmark. We also perform an ablation study to discover the performance contribution from different model layers. We show our ImageNet model generalizes well to other datasets: when the softmax classifier is retrained, it convincingly beats the current state-of-the-art results on Caltech-101 and Caltech-256 datasets. <s> BIB002 </s> Deep Learning Models for Retinal Blood Vessels Segmentation: A Review <s> 1) CONVOLUTIONAL LAYER <s> State-of-the-art models for semantic segmentation are based on adaptations of convolutional networks that had originally been designed for image classification. However, dense prediction and image classification are structurally different. In this work, we develop a new convolutional network module that is specifically designed for dense prediction. The presented module uses dilated convolutions to systematically aggregate multi-scale contextual information without losing resolution. The architecture is based on the fact that dilated convolutions support exponential expansion of the receptive field without loss of resolution or coverage. We show that the presented context module increases the accuracy of state-of-the-art semantic segmentation systems. In addition, we examine the adaptation of image classification networks to dense prediction and show that simplifying the adapted network can increase accuracy. <s> BIB003
Convolutional layers are the main building blocks of the CNNs. They learn the feature representation of the input images by performing convolutions over the inputs. The convolutional layer consists of several kernels which are used to compute different features from the input images. They ensure that the local connectivity-neurons are connected to a small region of the input which is known as the receptive field. The extracted feature maps are calculated by convolving the input with the kernels and then add the bias parameters to the feature. The convolutional layer has many kernels, and they are applied to the input image to calculate the output feature map. Each kernel is shared by all special locations of the input. The advantage of weight sharing is to reduce the complexity of the model and the training process of the network becomes easier. Mathematically, consider x as the input image, W is the kernel, and b is the bias for the convolutional layer. The feature map z generated from this layer is calculated as: Many researchers proposed different types of convolutional layers to improve the feature representation and to learn some kind of invariance. Tiled CNN  is one kind of the enhanced convolutional layer that tiles and multiplied different feature maps learn rotation and scale invariant features. Dilated CNN BIB003 is another recent development of the standard CNN, where more hyper-parameters are introduced to the convolutional layer. This strategy enhances the performance for tasks which need a large receptive field when they make the prediction, such as scene segmentation and speech synthesis and recognition. Also, there are other improved convolutional layers such as Network in Network (NIN) BIB001 for classification task which replaces the normal convolutional layer with multilayer perceptron convolution layer (mlpconv). It is like a micro neural network with more complex structures rather than conventional convolutional layer that uses linear filters followed by a nonlinear activation function. NIN enhances the model discriminability for local patches within the receptive field for classification task. Transposed Convolution BIB002 is another type of convolutional layer that is used to reconstruct the input again in regression tasks such as image segmentation, depth estimation, visualization, and image super-resolution. In the literature, this layer is called deconvolutional layer. The deconvolutional layer first upsamples the input to a specified factor and then performs a normal convolution operation on the upsampled result. Figure 7 illustrates the convolution operation.