A Survey on Deep Learning in Medical Image Analysis <s> Multi-stream architectures <s> Scene labeling consists of labeling each pixel in an image with the category of the object it belongs to. We propose a method that uses a multiscale convolutional network trained from raw pixels to extract dense feature vectors that encode regions of multiple sizes centered on each pixel. The method alleviates the need for engineered features, and produces a powerful representation that captures texture, shape, and contextual information. We report results using multiple postprocessing methods to produce the final labeling. Among those, we propose a technique to automatically retrieve, from a pool of segmentation components, an optimal set of components that best explain the scene; these components are arbitrary, for example, they can be taken from a segmentation tree or from any family of oversegmentations. The system yields record accuracies on the SIFT Flow dataset (33 classes) and the Barcelona dataset (170 classes) and near-record accuracy on Stanford background dataset (eight classes), while being an order of magnitude faster than competing approaches, producing a 320×240 image labeling in less than a second, including feature extraction. <s> BIB001 </s> A Survey on Deep Learning in Medical Image Analysis <s> Multi-stream architectures <s> Segmentation of anatomical structures in medical images is often based on a voxel/pixel classification approach. Deep learning systems, such as convolutional neural networks (CNNs), can infer a hierarchical representation of images that fosters categorization. We propose a novel system for voxel classification integrating three 2D CNNs, which have a one-to-one association with the xy, yz and zx planes of 3D image, respectively. We applied our method to the segmentation of tibial cartilage in low field knee MRI scans and tested it on 114 unseen scans. Although our method uses only 2D features at a single scale, it performs better than a state-of-the-art method using 3D multi-scale features. In the latter approach, the features and the classifier have been carefully adapted to the problem at hand. That we were able to get better results by a deep learning architecture that autonomously learns the features from the images is the main insight of this study. <s> BIB002 </s> A Survey on Deep Learning in Medical Image Analysis <s> Multi-stream architectures <s> In this paper, a multiscale convolutional network (MSCN) and graph-partitioning-based method is proposed for accurate segmentation of cervical cytoplasm and nuclei. Specifically, deep learning via the MSCN is explored to extract scale invariant features, and then, segment regions centered at each pixel. The coarse segmentation is refined by an automated graph partitioning method based on the pretrained feature. The texture, shape, and contextual information of the target objects are learned to localize the appearance of distinctive boundary, which is also explored to generate markers to split the touching nuclei. For further refinement of the segmentation, a coarse-to-fine nucleus segmentation framework is developed. The computational complexity of the segmentation is reduced by using superpixel instead of raw pixels. Extensive experimental results demonstrate that the proposed cervical nucleus cell segmentation delivers promising results and outperforms existing methods. <s> BIB003 </s> A Survey on Deep Learning in Medical Image Analysis <s> Multi-stream architectures <s> We propose a novel Computer-Aided Detection (CAD) system for pulmonary nodules using multi-view convolutional networks (ConvNets), for which discriminative features are automatically learnt from the training data. The network is fed with nodule candidates obtained by combining three candidate detectors specifically designed for solid, subsolid, and large nodules. For each candidate, a set of 2-D patches from differently oriented planes is extracted. The proposed architecture comprises multiple streams of 2-D ConvNets, for which the outputs are combined using a dedicated fusion method to get the final classification. Data augmentation and dropout are applied to avoid overfitting. On 888 scans of the publicly available LIDC-IDRI dataset, our method reaches high detection sensitivities of 85.4% and 90.1% at 1 and 4 false positives per scan, respectively. An additional evaluation on independent datasets from the ANODE09 challenge and DLCST is performed. We showed that the proposed multi-view ConvNets is highly suited to be used for false positive reduction of a CAD system. <s> BIB004 </s> A Survey on Deep Learning in Medical Image Analysis <s> Multi-stream architectures <s> HIGHLIGHTSAn efficient 11‐layers deep, multi‐scale, 3D CNN architecture.A novel training strategy that significantly boosts performance.The first employment of a 3D fully connected CRF for post‐processing.State‐of‐the‐art performance on three challenging lesion segmentation tasks.New insights into the automatically learned intermediate representations. ABSTRACT We propose a dual pathway, 11‐layers deep, three‐dimensional Convolutional Neural Network for the challenging task of brain lesion segmentation. The devised architecture is the result of an in‐depth analysis of the limitations of current networks proposed for similar applications. To overcome the computational burden of processing 3D medical scans, we have devised an efficient and effective dense training scheme which joins the processing of adjacent image patches into one pass through the network while automatically adapting to the inherent class imbalance present in the data. Further, we analyze the development of deeper, thus more discriminative 3D CNNs. In order to incorporate both local and larger contextual information, we employ a dual pathway architecture that processes the input images at multiple scales simultaneously. For post‐processing of the network's soft segmentation, we use a 3D fully connected Conditional Random Field which effectively removes false positives. Our pipeline is extensively evaluated on three challenging tasks of lesion segmentation in multi‐channel MRI patient data with traumatic brain injuries, brain tumours, and ischemic stroke. We improve on the state‐of‐the‐art for all three applications, with top ranking performance on the public benchmarks BRATS 2015 and ISLES 2015. Our method is computationally efficient, which allows its adoption in a variety of research and clinical settings. The source code of our implementation is made publicly available. <s> BIB005 </s> A Survey on Deep Learning in Medical Image Analysis <s> Multi-stream architectures <s> Automatic segmentation in MR brain images is important for quantitative analysis in large-scale studies with images acquired at all ages. This paper presents a method for the automatic segmentation of MR brain images into a number of tissue classes using a convolutional neural network. To ensure that the method obtains accurate segmentation details as well as spatial consistency, the network uses multiple patch sizes and multiple convolution kernel sizes to acquire multi-scale information about each voxel. The method is not dependent on explicit features, but learns to recognise the information that is important for the classification based on training data. The method requires a single anatomical MR image only. The segmentation method is applied to five different data sets: coronal T2-weighted images of preterm infants acquired at 30 weeks postmenstrual age (PMA) and 40 weeks PMA, axial T2- weighted images of preterm infants acquired at 40 weeks PMA, axial T1-weighted images of ageing adults acquired at an average age of 70 years, and T1-weighted images of young adults acquired at an average age of 23 years. The method obtained the following average Dice coefficients over all segmented tissue classes for each data set, respectively: 0.87, 0.82, 0.84, 0.86 and 0.91. The results demonstrate that the method obtains accurate segmentations in all five sets, and hence demonstrates its robustness to differences in age and acquisition protocol. <s> BIB006 </s> A Survey on Deep Learning in Medical Image Analysis <s> Multi-stream architectures <s> Suppression of bony structures in chest radiographs (CXRs) is potentially useful for radiologists and computer-aided diagnostic schemes. In this paper, we present an effective deep learning method for bone suppression in single conventional CXR using deep convolutional neural networks (ConvNets) as basic prediction units. The deep ConvNets were adapted to learn the mapping between the gradients of the CXRs and the corresponding bone images. We propose a cascade architecture of ConvNets (called CamsNet) to refine progressively the predicted bone gradients in which the ConvNets work at successively increased resolutions. The predicted bone gradients at different scales from the CamsNet are fused in a maximum-a-posteriori framework to produce the final estimation of a bone image. This estimation of a bone image is subtracted from the original CXR to produce a soft-tissue image in which the bone components are eliminated. Our method was evaluated on a dataset that consisted of 504 cases of real two-exposure dual-energy subtraction chest radiographs (404 cases for training and 100 cases for test). The results demonstrate that our method can produce high-quality and high-resolution bone and soft-tissue images. The average relative mean absolute error of the produced bone images and peak signal-to-noise ratio of the produced soft-tissue images were 3.83% and 38.7dB, respectively. The average bone suppression ratio of our method was 83.8% for the CXRs with pixel sizes of nearly 0.194mm. Furthermore, we apply the trained CamsNet model on the CXRs acquired by various types of X-ray machines, including scanned films, and our method can also produce visually appealing bone and soft-tissue images. <s> BIB007
The default CNN architecture can easily accommodate multiple sources of information or representations of the input, in the form of channels presented to the input layer. This idea can be taken further and channels can be merged at any point in the network. Under the intuition that different tasks require different ways of fusion, multi-stream architectures are being explored. These models, also referred to as dual pathway architectures BIB005 , have two main applications at the time of writing: (1) multi-scale image analysis and (2) 2.5D classification; both relevant for medical image processing tasks. For the detection of abnormalities, context is often an important cue. The most straightforward way to increase context is to feed larger patches to the network, but this can significantly increase the amount of parameters and memory requirements of a network. Consequently, architectures have been investigated where context is added in a down-scaled representation in addition to high resolution local information. To the best of our knowledge, the multi-stream multi-scale architecture was first explored by BIB001 , who used it for segmentation in natural images. Several medical applications have also successfully used this concept BIB005 BIB006 BIB003 BIB007 . As so much methodology is still developed on natural images, the challenge of applying deep learning techniques to the medical domain often lies in adapting existing architectures to, for instance, different input formats such as three-dimensional data. In early applications of CNNs to such volumetric data, full 3D convolutions and the resulting large amount of parameters were circumvented by dividing the Volume of Interest (VOI) into slices which are fed as different streams to a network. BIB002 were the first to use this approach for knee cartilage segmentation. Similarly, the network can be fed with multiple angled patches from the 3D-space in a multi-stream fashion, which has been applied by various authors in the context of medical imaging BIB004 . These approaches are also referred to as 2.5D classification.