A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters. This chapter is meant as a practical guide with recommendations for some of the most commonly used hyperparameters, in particular in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. It also discusses how to deal with the fact that more interesting results can be obtained when allowing one to adjust many hyper-parameters. Overall, it describes elements of the practice used to successfully and efficiently train and debug large-scale and often deep multi-layer neural networks. It closes with open questions about the training difficulties observed with deeper architectures. <s> BIB001 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> The use of machine learning algorithms frequently involves careful tuning of learning parameters and model hyperparameters. Unfortunately, this tuning is often a "black art" requiring expert experience, rules of thumb, or sometimes brute-force search. There is therefore great appeal for automatic approaches that can optimize the performance of any given learning algorithm to the problem at hand. In this work, we consider this problem through the framework of Bayesian optimization, in which a learning algorithm's generalization performance is modeled as a sample from a Gaussian process (GP). We show that certain choices for the nature of the GP, such as the type of kernel and the treatment of its hyperparameters, can play a crucial role in obtaining a good optimizer that can achieve expertlevel performance. We describe new algorithms that take into account the variable cost (duration) of learning algorithm experiments and that can leverage the presence of multiple cores for parallel experimentation. We show that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization for many algorithms including latent Dirichlet allocation, structured SVMs and convolutional neural networks. <s> BIB002 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net . <s> BIB003 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> Recently, deep learning has demonstrated great success in computer vision with the capability to learn powerful image features from a large training set. However, most of the published work has been confined to solving 2D problems, with a few limited exceptions that treated the 3D space as a composition of 2D orthogonal planes. The challenge of 3D deep learning is due to a much larger input vector, compared to 2D, which dramatically increases the computation time and the chance of over-fitting, especially when combined with limited training samples hundreds to thousands, typical for medical imaging applications. To address this challenge, we propose an efficient and robust deep learning algorithm capable of full 3D detection in volumetric data. A two-step approach is exploited for efficient detection. A shallow network with one hidden layer is used for the initial testing of all voxels to obtain a small number of promising candidates, followed by more accurate classification with a deep network. In addition, we propose two approaches, i.e., separable filter decomposition and network sparsification, to speed up the evaluation of a network. To mitigate the over-fitting issue, thereby increasing detection robustness, we extract small 3D patches from a multi-resolution image pyramid. The deeply learned image features are further combined with Haar wavelet features to increase the detection accuracy. The proposed method has been quantitatively evaluated for carotid artery bifurcation detection on a head-neck CT dataset from 455 patients. Compared to the state-of-the-art, the mean error is reduced by more than half, from 5.97 mm to 2.64 mm, with a detection speed of less than 1 s/volume. <s> BIB004 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> We address the task of object recognition in obstetric ultrasound videos using deep Convolutional Neural Networks (CNNs). A transfer learning based design is presented to study the transferability of features learnt from natural images to ultrasound image object recognition which on the surface is a very different problem. Our results demonstrate that CNNs initialised with large-scale pre-trained networks outperform those directly learnt from small-scale ultrasound data (91.5% versus 87.9%), in terms of object identification. <s> BIB005 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> Fetal development is noninvasively assessed by measuring the size of different structures in ultrasound (US) images. The reliability of these measurements is dependent upon the identification of the correct anatomical viewing plane, each of which contains different fetal structures. However, the automatic classification of the anatomical planes in fetal US images is challenging due to a number of factors, such as low signal-to-noise-ratios and the small size of the fetus. Current approaches for plane classification are limited to simpler subsets of the problem: only classifying planes within specific body regions or using temporal information from videos. In this paper, we propose a new general method for the classification of anatomical planes in fetal US images. Our method trains two convolutional neural networks to learn the best US and saliency features. The fusion of these features overcomes the challenges associated with US fetal imaging by emphasising the salient features within US images that best discriminate different planes. Our method achieved higher classification accuracy than a state-of-the-art baseline for 12 of the 13 different planes found in a clinical dataset of fetal US images. <s> BIB006 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> The availability of training data for supervision is a frequently encountered bottleneck of medical image analysis methods. While typically established by a clinical expert rater, the increase in acquired imaging data renders traditional pixel-wise segmentations less feasible. In this paper, we examine the use of a crowdsourcing platform for the distribution of super-pixel weak annotation tasks and collect such annotations from a crowd of non-expert raters. The crowd annotations are subsequently used for training a fully convolutional neural network to address the problem of fetal brain segmentation in T2-weighted MR images. Using this approach we report encouraging results compared to highly targeted, fully supervised methods and potentially address a frequent problem impeding image analysis research. <s> BIB007 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> In this paper, we propose a hybrid approach combining traditional texture analysis methods with deep learning for the automatic detection and measurement of abdominal contour from 2-D fetal ultrasound images. Following a learning-based procedure for region of interest (ROI) localization to segment the abdominal boundary, we show that convolutional neural networks (CNNs) outperform other state-of-the-art texture features and conventional classifiers, in addressing the binary classification problem of distinguishing between abdomen versus non-abdomen regions. However, we obtain significantly better segmentation results in identifying the best ROI containing fetal abdomen, when the predictions from CNN are combined with those from gradient boosting machine (GBM) using histogram of oriented gradient (HOG) features. We trained our method on a set of 70 images and tested them on another distinct set of 70 images. We obtained a mean DICE similarity coefficient of 0.90, which shows excellent overlap with the ground truth. We report that the mean computed gestational age difference between our segmentation results and the ground truth, is within two weeks for 90% (and within one week for 70%) of the testing cases. <s> BIB008 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> Recently, magnetic resonance imaging has revealed to be important for the evaluation of placenta’s health during pregnancy. Quantitative assessment of the placenta requires a segmentation, which proves to be challenging because of the high variability of its position, orientation, shape and appearance. Moreover, image acquisition is corrupted by motion artifacts from both fetal and maternal movements. In this paper we propose a fully automatic segmentation framework of the placenta from structural T2-weighted scans of the whole uterus, as well as an extension in order to provide an intuitive pre-natal view into this vital organ. We adopt a 3D multi-scale convolutional neural network to automatically identify placental candidate pixels. The resulting classification is subsequently refined by a 3D dense conditional random field, so that a high resolution placental volume can be reconstructed from multiple overlapping stacks of slices. Our segmentation framework has been tested on 66 subjects at gestational ages 20–38 weeks achieving a Dice score of \(71.95\pm 19.79\,\%\) for healthy fetuses with a fixed scan sequence and \(66.89\pm 15.35\,\%\) for a cohort mixed with cases of intrauterine fetal growth restriction using varying scan parameters. <s> BIB009 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> During the last years Deep Learning and especially Convolutional Neural Networks (CNN) have set new standards for different computer vision tasks like image classification and semantic segmentation. In this paper, a CNN for 3D volume segmentation based on recently introduced deep learning components will be presented. In addition to using image patches as input for a CNN, the usage of orthogonal patches, which combine shape and locality information with intensity information for CNN training will be evaluated. For this purpose a publically available CT dataset of the head-neck region has been used and the results have been compared with other state-of-the art atlas- and model-based segmentation approaches. <s> BIB010 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> Studying tongue motion during speech using ultrasound is a standard procedure, but automatic ultrasound image labelling remains a challenge, as standard tongue shape extraction methods typically require human intervention. This article presents a method based on deep neural networks to automatically extract tongue contour from ultrasound images on a speech dataset. We use a deep autoencoder trained to learn the relationship between an image and its related contour, so that the model is able to automatically reconstruct contours from the ultrasound image alone. In this paper, we use an automatic labelling algorithm instead of time-consuming hand-labelling during the training process, and estimate the performances of both automatic labelling and contour extraction as compared to hand-labelling. Observed results show quality scores comparable to the state of the art. <s> BIB011 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> We explore the applicability of deep convolutional neural networks (CNNs) for multiple landmark localization in medical image data. Exploiting the idea of regressing heatmaps for individual landmark locations, we investigate several fully convolutional 2D and 3D CNN architectures by training them in an end-to-end manner. We further propose a novel SpatialConfiguration-Net architecture that effectively combines accurate local appearance responses with spatial landmark configurations that model anatomical variation. Evaluation of our different architectures on 2D and 3D hand image datasets show that heatmap regression based on CNNs achieves state-of-the-art landmark localization performance, with SpatialConfiguration-Net being robust even in case of limited amounts of training data. <s> BIB012 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> Point of care diagnostics using microscopy and computer vision methods have been applied to a number of practical problems, and are particularly relevant to low-income, high disease burden areas. However, this is subject to the limitations in sensitivity and specificity of the computer vision methods used. In general, deep learning has recently revolutionised the field of computer vision, in some cases surpassing human performance for other object recognition tasks. In this paper, we evaluate the performance of deep convolutional neural networks on three different microscopy tasks: diagnosis of malaria in thick blood smears, tuberculosis in sputum samples, and intestinal parasite eggs in stool samples. In all cases accuracy is very high and substantially better than an alternative approach more representative of traditional medical imaging techniques. <s> BIB013 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> Deep convolutional neural networks have achieved great results on image classification problems. In this paper, a new method using a deep convolutional neural network for detecting blood vessels in B-mode ultrasound images is presented. Automatic blood vessel detection may be useful in medical applications such as deep venous thrombosis detection, anesthesia guidance and catheter placement. The proposed method is able to determine the position and size of the vessels in images in real-time. 12,804 subimages of the femoral region from 15 subjects were manually labeled. Leave-one-subject-out cross validation was used giving an average accuracy of 94.5 %, a major improvement from previous methods which had an accuracy of 84 % on the same dataset. The method was also validated on a dataset of the carotid artery to show that the method can generalize to blood vessels on other regions of the body. The accuracy on this dataset was 96 %. <s> BIB014 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> To improve the diagnostic accuracy of cervical dysplasia, it is important to fuse multimodal information collected during a patient’s screening visit. However, current multimodal frameworks suffer from low sensitivity at high specificity levels, due to their limitations in learning correlations among highly heterogeneous modalities. In this paper, we design a deep learning framework for cervical dysplasia diagnosis by leveraging multimodal information. We first employ the convolutional neural network (CNN) to convert the low-level image data into a feature vector fusible with other non-image modalities. We then jointly learn the non-linear correlations among all modalities in a deep neural network. Our multimodal framework is an end-to-end deep network which can learn better complementary features from the image and non-image modalities. It automatically gives the final diagnosis for cervical dysplasia with 87.83 % sensitivity at 90 % specificity on a large dataset, which significantly outperforms methods using any single source of information alone and previous multimodal frameworks. <s> BIB015 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> Limited angle problem is a challenging issue in x-ray computed tomography (CT) field. Iterative reconstruction methods that utilize the additional prior can suppress artifacts and improve image quality, but unfortunately require increased computation time. An interesting way is to restrain the artifacts in the images reconstructed from the practical filtered back projection (FBP) method. Frikel and Quinto have proved that the streak artifacts in FBP results could be characterized. It indicates that the artifacts created by FBP method have specific and similar characteristics in a stationary limited-angle scanning configuration. Based on this understanding, this work aims at developing a method to extract and suppress specific artifacts of FBP reconstructions for limited-angle tomography. A data-driven learning-based method is proposed based on a deep convolutional neural network. An end-to-end mapping between the FBP and artifact-free images is learned and the implicit features involving artifacts will be extracted and suppressed via nonlinear mapping. The qualitative and quantitative evaluations of experimental results indicate that the proposed method show a stable and prospective performance on artifacts reduction and detail recovery for limited angle tomography. The presented strategy provides a simple and efficient approach for improving image quality of the reconstruction results from limited projection data. <s> BIB016 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> In this paper, we propose DeepCut, a method to obtain pixelwise object segmentations given an image dataset labelled with bounding box annotations. It extends the approach of the well-known GrabCut method to include machine learning by training a neural network classifier from bounding box annotations. We formulate the problem as an energy minimisation problem over a densely-connected conditional random field and iteratively update the training targets to obtain pixelwise object segmentations. Additionally, we propose variants of the DeepCut method and compare those to a naive approach to CNN training under weak supervision. We test its applicability to solve brain and lung segmentation problems on a challenging fetal magnetic resonance dataset and obtain encouraging results in terms of accuracy. <s> BIB017 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> Segmentation of fetal left ventricle (LV) in echocardiographic sequences is important for further quantitative analysis of fetal cardiac function. However, image gross inhomogeneities and fetal random movements make the segmentation a challenging problem. In this paper, a dynamic convolutional neural networks (CNN) based on multiscale information and fine-tuning is proposed for fetal LV segmentation. The CNN is pretrained by amount of labeled training data. In the segmentation, the first frame of each echocardiographic sequence is delineated manually. The dynamic CNN is fine-tuned by deep tuning with the first frame and shallow tuning with the rest of frames, respectively, to adapt to the individual fetus. Additionally, to separate the connection region between LV and left atrium (LA), a matching approach, which consists of block matching and line matching, is used for mitral valve (MV) base points tracking. Advantages of our proposed method are compared with an active contour model (ACM), a dynamical appearance model (DAM), and a fixed multiscale CNN method. Experimental results in 51 echocardiographic sequences show that the segmentation results agree well with the ground truth, especially in the cases with leakage, blurry boundaries, and subject-to-subject variations. The CNN architecture can be simple, and the dynamic fine-tuning is efficient. <s> BIB018 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> Characterization of carotid plaque composition, more specifically the amount of lipid core, fibrous tissue, and calcified tissue, is an important task for the identification of plaques that are prone to rupture, and thus for early risk estimation of cardiovascular and cerebrovascular events. Due to its low costs and wide availability, carotid ultrasound has the potential to become the modality of choice for plaque characterization in clinical practice. However, its significant image noise, coupled with the small size of the plaques and their complex appearance, makes it difficult for automated techniques to discriminate between the different plaque constituents. In this paper, we propose to address this challenging problem by exploiting the unique capabilities of the emerging deep learning framework. More specifically, and unlike existing works which require a priori definition of specific imaging features or thresholding values, we propose to build a convolutional neural network (CNN) that will automatically extract from the images the information that is optimal for the identification of the different plaque constituents. We used approximately 90 000 patches extracted from a database of images and corresponding expert plaque characterizations to train and to validate the proposed CNN. The results of cross-validation experiments show a correlation of about 0.90 with the clinical assessment for the estimation of lipid core, fibrous cap, and calcified tissue areas, indicating the potential of deep learning for the challenging task of automatic characterization of plaque composition in carotid ultrasound. <s> BIB019 </s> A Survey on Deep Learning in Medical Image Analysis <s> Key aspects of successful deep learning methods <s> In ultrasound images, most thyroid nodules are in heterogeneous appearances with various internal components and also have vague boundaries, so it is difficult for physicians to discriminate malignant thyroid nodules from benign ones. In this study, we propose a hybrid method for thyroid nodule diagnosis, which is a fusion of two pre-trained convolutional neural networks (CNNs) with different convolutional layers and fully-connected layers. Firstly, the two networks pre-trained with ImageNet database are separately trained. Secondly, we fuse feature maps learned by trained convolutional filters, pooling and normalization operations of the two CNNs. Finally, with the fused feature maps, a softmax classifier is used to diagnose thyroid nodules. The proposed method is validated on 15,000 ultrasound images collected from two local hospitals. Experiment results show that the proposed CNN based methods can accurately and effectively diagnose thyroid nodules. In addition, the fusion of the two CNN based models lead to significant performance improvement, with an accuracy of 83.02%±0.72%. These demonstrate the potential clinical applications of this method. <s> BIB020
After reviewing so many papers one would expect to be able to distill the perfect deep learning method and architecture for each individual task and application area. Although convolutional neural networks (and derivatives) are now clearly the top performers in most medical image analysis competitions, one striking conclusion we can draw is that the exact architecture is not the most important determinant in getting a good solution. We have seen, for example in challenges like the Kaggle Diabetic Retinopathy Challenge, that many researchers use the exact same architectures, the same type of networks, but have widely varying results. A key aspect that is often overlooked is that expert knowledge about the task to be solved can provide advantages that go beyond adding more layers to a CNN. Groups and researchers that obtain good performance when applying deep learning algorithms often differentiate themselves in aspects outside of the deep network, like novel data preprocessing or augmentation techniques. An example is that the best performing method in the CAMELYON16-challenge improved significantly (AUC from 0.92 to 0.99) by adding a stain normalization pre-processing step to improve generalization without changing the CNN. Other papers focus on data augmentation strategies to make networks more robust, and they report that these strategies are essential to obtain good performance. An example is the elastic deformations that were applied in the original U-Net paper BIB003 . Augmentation and pre-processing are, of course, not the only key contributors to good solutions. Several researchers have shown that designing architectures incorporating unique task-specific properties can obtain better results than straightforward CNNs. Two examples which we encountered several times are multi-view BIB005 Frame labeling US CNN 4 class frame classification using transfer learning with pre-trained networks BIB006 Frame labeling US CNN 12 standard anatomical planes, CNN extracts features for support vector machine BIB007 Segmentation with non expert labels MRI CNN Crowd-sourcing annotation efforts to segment brain structures BIB017 Segmentation given bounding box MRI CNN CNN and CRF for segmentation of structures BIB008 Quantification US CNN Hybrid system using CNN and texture features to find abdominal circumference BIB018 Left ventricle segmentation US CNN Frame-by-frame segmentation by dynamically fine-tuning CNN to the latest frame Wound segmentation photographs CNN Additional detection of infection risk and healing progress  Chemotherapy response prediction PET CNN CNN outperforms classical radiomics features in patients with esophageal cancer BIB004 Carotid artery bifurcation detection CT CNN Two stage detection process, CNNs combined with Haar features BIB009 Placenta segmentation MRI CNN 3D multi-stream CNN with extension for motion correction BIB010 Head&Neck tumor segmentation CT CNN 3 orthogonal patches in 2D CNNs, combined with other features Jaumard- BIB011 Tongue contour extraction US RBM Analysis of tongue motion during speech, combines auto-encoders with RBMs BIB012 Hand landmark detection X-ray CNN Various architectures are compared BIB013 Disease detection microscopy CNN Smartphone mounted on microscope detects malaria, tuberculosis & parasite eggs BIB014 Vessel detection and segmentation US CNN Femoral and carotid vessels analyzed with standard fCNN  Task recognition in laparoscopy Videos CNN Fine-tuned AlexNet applied to video frames BIB015 Cervical dysplasia cervigrams CNN Fine-tuned pre-trained network with added non-imaging features  Esophageal microvessel classification Microscopy CNN Simple CNN used for feature extraction BIB016 Image reconstruction CT CNN Reconstructing from limited angle measurements, reducing reconstruction artefacts BIB019 Carotid plaque classification US CNN Simple CNN for characterization of carotid plaque composition in ultrasound BIB020 Thyroid nodule detection US CNN CNN and standard features combines for 2D US analysis and multi-scale networks. Other, often underestimated, parts of network design are the network input size and receptive field (i.e. the area in input space that contributes to a single output unit). Input sizes should be selected considering for example the required resolution and context to solve a problem. One might increase the size of the patch to obtain more context, but without changing the receptive field of the network this might not be beneficial. As a standard sanity check researchers could perform the same task themselves via visual assessment of the network input. If they, or domain experts, cannot achieve good performance, the chance that you need to modify your network input or architecture is high. The last aspect we want to touch on is model hyperparameter optimization (e.g. learning rate, dropout rate), which can help squeeze out extra performance from a network. We believe this is of secondary im-portance with respect to performance to the previously discussed topics and training data quality. Disappointingly, no clear recipe can be given to obtain the best set of hyper-parameters as it is a highly empirical exercise. Most researchers fall back to an intuition-based random search , which often seems to work well enough. Some basic tips have been covered before by BIB001 . Researchers have also looked at Bayesian methods for hyper-parameter optimization BIB002 , but this has not been applied in medical image analysis as far as we are aware of.