A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> We address a central problem of neuroanatomy, namely, the automatic segmentation of neuronal structures depicted in stacks of electron microscopy (EM) images. This is necessary to efficiently map 3D brain structure and connectivity. To segment biological neuron membranes, we use a special type of deep artificial neural network as a pixel classifier. The label of each pixel (membrane or non-membrane) is predicted from raw pixel values in a square window centered on it. The input layer maps each window pixel to a neuron. It is followed by a succession of convolutional and max-pooling layers which preserve 2D information and extract features with increasing levels of abstraction. The output layer produces a calibrated probability for each class. The classifier is trained by plain gradient descent on a 512 × 512 × 30 stack with known ground truth, and tested on a stack of the same size (ground truth unknown to the authors) by the organizers of the ISBI 2012 EM Segmentation Challenge. Even without problem-specific postprocessing, our approach outperforms competing techniques by a large margin in all three considered metrics, i.e. rand error, warping error and pixel error. For pixel error, our approach is the only one outperforming a second human observer. <s> BIB001 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Background: The mitotic figure recognition contest at the 2012 International Conference on Pattern Recognition (ICPR) challenges a system to identify all mitotic figures in a region of interest of hematoxylin and eosin stained tissue, using each of three scanners (Aperio, Hamamatsu, and multispectral). Methods: Our approach combines manually designed nuclear features with the learned features extracted by convolutional neural networks (CNN). The nuclear features capture color, texture, and shape information of segmented regions around a nucleus. The use of a CNN handles the variety of appearances of mitotic figures and decreases sensitivity to the manually crafted features and thresholds. Results : On the test set provided by the contest, the trained system achieves F1 scores up to 0.659 on color scanners and 0.589 on multispectral scanner. Conclusions : We demonstrate a powerful technique combining segmentation-based features with CNN, identifying the majority of mitotic figures with a fair precision. Further, we show that the approach accommodates information from the additional focal planes and spectral bands from a multi-spectral scanner without major redesign. <s> BIB002 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> This paper studies the effectiveness of accomplishing high-level tasks with a minimum of manual annotation and good feature representations for medical images. In medical image analysis, objects like cells are characterized by significant clinical features. Previously developed features like SIFT and HARR are unable to comprehensively represent such objects. Therefore, feature representation is especially important. In this paper, we study automatic extraction of feature representation through deep learning (DNN). Furthermore, detailed annotation of objects is often an ambiguous and challenging task. We use multiple instance learning (MIL) framework in classification training with deep learning features. Several interesting conclusions can be drawn from our work: (1) automatic feature learning outperforms manual feature; (2) the unsupervised approach can achieve performance that's close to fully supervised approach (93.56%) vs. (94.52%); and (3) the MIL performance of coarse label (96.30%) outweighs the supervised performance of fine label (95.40%) in supervised deep learning features. <s> BIB003 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Counting bacterial colonies on microbiological culture plates is a time-consuming, error-prone, nevertheless fundamental task in microbiology. Computer vision based approaches can increase the efficiency and the reliability of the process, but accurate counting is challenging, due to the high degree of variability of agglomerated colonies. In this paper, we propose a solution which adopts Convolutional Neural Networks (CNN) for counting the number of colonies contained in confluent agglomerates, that scored an overall accuracy of the 92.8% on a large challenging dataset. The proposed CNN-based technique for estimating the cardinality of colony aggregates outperforms traditional image processing approaches, becoming a promising approach to many related applications. <s> BIB004 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net . <s> BIB005 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> We propose a semi-automated pipeline for the detection of possible cell divisions in live-imaging microscopy and the classification of these mitosis candidates using a Convolutional Neural Network (CNN). We use time-lapse images of NIH3T3 scratch assay cultures, extract patches around bright candidate regions that then undergo segmentation and binarization, followed by a classification of the binary patches into either containing or not containing cell division. The classification is performed by training a Convolutional Neural Network on a specially constructed database. We show strong results of AUC = 0.91 and F-score = 0.89, competitive with state-of-the-art methods in this field. <s> BIB006 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> In this paper, a multiscale convolutional network (MSCN) and graph-partitioning-based method is proposed for accurate segmentation of cervical cytoplasm and nuclei. Specifically, deep learning via the MSCN is explored to extract scale invariant features, and then, segment regions centered at each pixel. The coarse segmentation is refined by an automated graph partitioning method based on the pretrained feature. The texture, shape, and contextual information of the target objects are learned to localize the appearance of distinctive boundary, which is also explored to generate markers to split the touching nuclei. For further refinement of the segmentation, a coarse-to-fine nucleus segmentation framework is developed. The computational complexity of the segmentation is reduced by using superpixel instead of raw pixels. Extensive experimental results demonstrate that the proposed cervical nucleus cell segmentation delivers promising results and outperforms existing methods. <s> BIB007 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Robust and accurate nuclei localization in microscopy image can provide crucial clues for accurate computer-aid diag In this paper, we propose a convolutional neural network (CNN) based hough voting method to localize nucleus centroids with heavy cluttering and morphologic variations in microscopy images. Our method, which we name as deep voting, mainly consists of two steps. (1) Given an input image, our method assigns each local patch several pairs of voting offset vectors which indicate the positions it votes to, and the corresponding voting confidence (used to weight each votes), our model can be viewed as an implicit hough-voting codebook. (2) We collect the weighted votes from all the testing patches and compute the final voting density map in a way similar to Parzen-window estimation. The final nucleus positions are identified by searching the local maxima of the density map. Our method only requires a few annotation efforts (just one click near the nucleus center). Experiment results on Neuroendocrine Tumor (NET) microscopy images proves the proposed method to be state-of-the-art. <s> BIB008 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Robust cell detection serves as a critical prerequisite for many biomedical image analysis applications. In this paper, we present a novel convolutional neural network (CNN) based structured regression model, which is shown to be able to handle touching cells, inhomogeneous background noises, and large variations in sizes and shapes. The proposed method only requires a few training images with weak annotations (just one click near the center of the object). Given an input image patch, instead of providing a single class label like many traditional methods, our algorithm will generate the structured outputs (referred to as proximity patches). These proximity patches, which exhibit higher values for pixels near cell centers, will then be gathered from all testing image patches and fused to obtain the final proximity map, where the maximum positions indicate the cell centroids. The algorithm is tested using three data sets representing different image stains and modalities. The comparative experiments demonstrate the superior performance of this novel method over existing state-of-the-art. <s> BIB009 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Segmentation of histopathology sections is an ubiquitous requirement in digital pathology and due to the large variability of biological tissue, machine learning techniques have shown superior performance over standard image processing methods. As part of the GlaS@MICCAI2015 colon gland segmentation challenge, we present a learning-based algorithm to segment glands in tissue of benign and malignant colorectal cancer. Images are preprocessed according to the Hematoxylin-Eosin staining protocol and two deep convolutional neural networks (CNN) are trained as pixel classifiers. The CNN predictions are then regularized using a figure-ground segmentation based on weighted total variation to produce the final segmentation result. On two test sets, our approach achieves a tissue classification accuracy of 98% and 94%, making use of the inherent capability of our system to distinguish between benign and malignant tissue. <s> BIB010 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Accurate cell segmentation is vital for the development of reliable microscopy image analysis methods. It is a very challenging problem due to low contrast, weak boundaries, and conjoined and overlapping cells; producing many ambiguous regions, which lower the performance of automated segmentation methods. Cell proposals provide an efficient way of exploiting both spatial and temporal context, which can be very helpful in many of these ambiguous regions. However, most proposal based microscopy image analysis methods rely on fairly simple proposal generation stage, limiting their performance. In this paper, we propose a convolutional neural network based method which provides cell segmentation proposals, which can be used for cell detection, segmentation and tracking. We evaluate our method on datasets from histology, fluorescence and phase contrast microscopy and show that it outperforms state of the art cell detection and segmentation methods. <s> BIB011 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> The lack of publicly available ground-truth data has been identified as the major challenge for transferring recent developments in deep learning to the biomedical imaging domain. Though crowdsourcing has enabled annotation of large scale databases for real world images, its application for biomedical purposes requires a deeper understanding and hence, more precise definition of the actual annotation task. The fact that expert tasks are being outsourced to non-expert users may lead to noisy annotations introducing disagreement between users. Despite being a valuable resource for learning annotation models from crowdsourcing, conventional machine-learning methods may have difficulties dealing with noisy annotations during training. In this manuscript, we present a new concept for learning from crowds that handle data aggregation directly as part of the learning process of the convolutional neural network (CNN) via additional crowdsourcing layer (AggNet). Besides, we present an experimental study on learning from crowds designed to answer the following questions. 1) Can deep CNN be trained with data collected from crowdsourcing? 2) How to adapt the CNN to train on multiple types of annotation datasets (ground truth and crowd-based)? 3) How does the choice of annotation and aggregation affect the accuracy? Our experimental setup involved Annot8, a self-implemented web-platform based on Crowdflower API realizing image annotation tasks for a publicly available biomedical image database. Our results give valuable insights into the functionality of deep CNN learning from crowd annotations and prove the necessity of data aggregation integration. <s> BIB012 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Accurate and robust cell nuclei classification is the cornerstone for a wider range of tasks in digital and Computational Pathology. However, most machine learning systems require extensive labeling from expert pathologists for each individual problem at hand, with no or limited abilities for knowledge transfer between datasets and organ sites. In this paper we implement and evaluate a variety of deep neural network models and model ensembles for nuclei classification in renal cell cancer (RCC) and prostate cancer (PCa). We propose a convolutional neural network system based on residual learning which significantly improves over the state-of-the-art in cell nuclei classification. However, the main thrust of our work is to demonstrate for the first time the models ability to transfer the learned concepts from one organ type to another, with robust performance. Finally, we show that the combination of tissue types during training increases not only classification accuracy but also overall survival analysis. The best model, trained on combined data of RCC and PCa, exhibits optimal performance on PCa classification and better survival group stratification than an expert pathologist ($p=0.006$). All code, image data and expert labels are made publicly available to serve as benchmark for the community for future research into computational pathology. <s> BIB013 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Mitosis counting is one of the strongest prognostic markers for invasive breast cancer diagnosis. Clinical visual examination on histology slides by pathologists is tedious, error-prone and time-consuming. Furthermore, with the advent of whole slide imaging for high-throughput digitization, a large quantity of histology images need to be analyzed. Therefore, automated mitosis detection methods are highly demanded in clinical practice. In this paper, we proposed a deep regression network (DRN) to meet these challenges. It consisted of a downsampling path for extracting the high level information and an upsampling path for outputting the score map with original input size, thus it can be trained in an end-to-end way. In addition, we transferred knowledge learned from cross domains to mitigate the issue of insufficient medical training data. Experimental results on the benchmark dataset 2012ICPR Mitosis Detection Challenge demonstrated the efficacy of our approach, which achieved comparable or better performance than the state-of-the-art methods. <s> BIB014 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Detection of tumor nuclei in cancer histology images requires sophisticated techniques due to the irregular shape, size and chromatin texture of the tumor nuclei. Some very recently proposed methods employ deep convolutional neural networks (CNNs) to detect cells in H&E stained images. However, all such methods use some form of raw pixel intensities as input and rely on the CNN to learn the deep features. In this work, we extend a recently proposed spatially constrained CNN (SC-CNN) by proposing features that capture texture characteristics and show that although CNN produces good results on automatically learned features, it can perform better if the input consists of a combination of handcrafted features and the raw data. The handcrafted features are computed through the scattering transform which gives non-linear invariant texture features. The combination of handcrafted features with raw data produces sharp proximity maps and better detection results than the results of raw intensities with a similar kind of CNN architecture. <s> BIB015 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> We propose a Hierarchical Convolution Neural Network (HCNN) for mitosis event detection in time-lapse phase contrast microscopy. Our method contains two stages: first, we extract candidate spatial-temporal patch sequences in the input image sequences which potentially contain mitosis events. Then, we identify if each patch sequence contains mitosis event or not using a hieratical convolutional neural network. In the experiments, we validate the design of our proposed architecture and evaluate the mitosis event detection performance. Our method achieves 99.1 % precision and 97.2 % recall in very challenging image sequences of multipolar-shaped C3H10T1/2 mesenchymal stem cells and outperforms other state-of-the-art methods. Furthermore, the proposed method does not depend on hand-crafted feature design or cell tracking. It can be straightforwardly adapted to event detection of other different cell types. <s> BIB016 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Mitochondrial functions are essential for cell survival. Pathologic situations, e.g. cancer, can impair mitochondrial function which is frequently reflected by an altered morphology. So far, feature description of mitochondrial structure in cancer remains largely qualitative. In this study, we propose a learning-based approach to quantitatively assess the structure of mitochondria isolated from liver tumor cell lines using convolutional neural network (CNN). Besides achieving a high classification accuracy on isolated mitochondria from healthy tissue and different tumor cell lines which the CNN model was trained on, CNN is also able to classify unseen tumor cell lines, which suggests its superior capability to capture the intrinsic structural transition from healthy to tumor mitochondria. <s> BIB017 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> The recognition of the staining patterns of Human Epithelial-2 (HEp-2) cells in indirect immunofluorescence (IIF) images is essential for the diagnosis of several autoimmune diseases. The main challenge is the extraction and selection of the optimal feature set that not only represents the cells' characteristics, but also distinguishes between the classes of cell images with similar appearances. In this paper, we propose a system to classify HEp-2 cell images by applying transfer learning from a pre-trained deep convolutional neural network (CNN) to extract the generic features and then using a feature selection method to get the most relevant features for classification. Although the CNN was trained with a dataset very different from cell images, our system is capable of extracting important semantic features that represent a HEp-2 cell image. When evaluated on the ICPR2012 cell dataset, our method outperforms all other methods on the dataset of the 2012 competition, and demonstrates stable performance under different test protocols. <s> BIB018 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Detection and classification of cell nuclei in histopathology images of cancerous tissue stained with the standard hematoxylin and eosin stain is a challenging task due to cellular heterogeneity. Deep learning approaches have been shown to produce encouraging results on histopathology images in various studies. In this paper, we propose a Spatially Constrained Convolutional Neural Network (SC-CNN) to perform nucleus detection. SC-CNN regresses the likelihood of a pixel being the center of a nucleus, where high probability values are spatially constrained to locate in the vicinity of the centers of nuclei. For classification of nuclei, we propose a novel Neighboring Ensemble Predictor (NEP) coupled with CNN to more accurately predict the class label of detected cell nuclei. The proposed approaches for detection and classification do not require segmentation of nuclei. We have evaluated them on a large dataset of colorectal adenocarcinoma images, consisting of more than 20,000 annotated nuclei belonging to four different classes. Our results show that the joint detection and classification of the proposed SC-CNN and NEP produces the highest average F1 score as compared to other recently published approaches. Prospectively, the proposed methods could offer benefit to pathology practice in terms of quantitative analysis of tissue constituents in whole-slide images, and potentially lead to a better understanding of cancer. <s> BIB019 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> The size of nuclei in histological preparations from excised breast tumors is predictive of patient outcome (large nuclei indicate poor outcome). Pathologists take into account nuclear size when performing breast cancer grading. In addition, the mean nuclear area (MNA) has been shown to have independent prognostic value. The straightforward approach to measuring nuclear size is by performing nuclei segmentation. We hypothesize that given an image of a tumor region with known nuclei locations, the area of the individual nuclei and region statistics such as the MNA can be reliably computed directly from the image data by employing a machine learning model, without the intermediate step of nuclei segmentation. Towards this goal, we train a deep convolutional neural network model that is applied locally at each nucleus location, and can reliably measure the area of the individual nuclei and the MNA. Furthermore, we show how such an approach can be extended to perform combined nuclei detection and measurement, which is reminiscent of granulometry. <s> BIB020 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Robust cell detection in histopathological images is a crucial step in the computer-assisted diagnosis methods. In addition, recent studies show that subtypes play an significant role in better characterization of tumor growth and outcome prediction. In this paper, we propose a novel subtype cell detection method with an accelerated deep convolution neural network. The proposed method not only detects cells but also gives subtype cell classification for the detected cells. Based on the subtype cell detection results, we extract subtype cell related features and use them in survival prediction. We demonstrate that our proposed method has excellent subtype cell detection performance and our proposed subtype cell features can achieve more accurate survival prediction. <s> BIB021 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Computer-aided image analysis of histopathology specimens could potentially provide support for early detection and improved characterization of diseases such as brain tumor, pancreatic neuroendocrine tumor (NET), and breast cancer. Automated nucleus segmentation is a prerequisite for various quantitative analyses including automatic morphological feature computation. However, it remains to be a challenging problem due to the complex nature of histopathology images. In this paper, we propose a learning-based framework for robust and automatic nucleus segmentation with shape preservation. Given a nucleus image, it begins with a deep convolutional neural network (CNN) model to generate a probability map, on which an iterative region merging approach is performed for shape initializations. Next, a novel segmentation algorithm is exploited to separate individual nuclei combining a robust selection-based sparse shape model and a local repulsive deformable model. One of the significant benefits of the proposed framework is that it is applicable to different staining histopathology images. Due to the feature learning characteristic of the deep CNN and the high level shape prior modeling, the proposed method is general enough to perform well across multiple scenarios. We have tested the proposed algorithm on three large-scale pathology image datasets using a range of different tissue and stain preparations, and the comparative experiments with recent state of the arts demonstrate the superior performance of the proposed approach. <s> BIB022 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Automated nuclear detection is a critical step for a number of computer assisted pathology related image analysis algorithms such as for automated grading of breast cancer tissue specimens. The Nottingham Histologic Score system is highly correlated with the shape and appearance of breast cancer nuclei in histopathological images. However, automated nucleus detection is complicated by 1) the large number of nuclei and the size of high resolution digitized pathology images, and 2) the variability in size, shape, appearance, and texture of the individual nuclei. Recently there has been interest in the application of "Deep Learning" strategies for classification and analysis of big image data. Histopathology, given its size and complexity, represents an excellent use case for application of deep learning strategies. In this paper, a Stacked Sparse Autoencoder (SSAE), an instance of a deep learning strategy, is presented for efficient nuclei detection on high-resolution histopathological images of breast cancer. The SSAE learns high-level features from just pixel intensities alone in order to identify distinguishing features of nuclei. A sliding window operation is applied to each image in order to represent image patches via high-level features obtained via the auto-encoder, which are then subsequently fed to a classifier which categorizes each image patch as nuclear or non-nuclear. Across a cohort of 500 histopathological images (2200 × 2200) and approximately 3500 manually segmented individual nuclei serving as the groundtruth, SSAE was shown to have an improved F-measure 84.49% and an average area under Precision-Recall curve (AveP) 78.83%. The SSAE approach also out-performed nine other state of the art nuclear detection strategies. <s> BIB023 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> In this paper, we present a generalized distributed deep neural network architecture to detect cells in whole-slide high-resolution histopathological images, which usually hold \(10^{8}\) to \(10^{10}\) pixels. Our framework can adapt and accelerate any deep convolutional neural network pixel-wise cell detector to perform whole-slide cell detection within a reasonable time limit. We accelerate the convolutional neural network forwarding through a sparse kernel technique, eliminating almost all of the redundant computation among connected patches. Since the disk I/O becomes a bottleneck when the image size scale grows larger, we propose an asynchronous prefetching technique to diminish a large portion of the disk I/O time. An unbalanced distributed sampling strategy is proposed to enhance the scalability and communication efficiency in distributed computing. Blending advantages of the sparse kernel, asynchronous prefetching and distributed sampling techniques, our framework is able to accelerate the conventional convolutional deep learning method by nearly 10, 000 times with same accuracy. Specifically, our method detects cells in a \(10^{8}\)-pixel (\(10^4\times 10^4\)) image in 20 s (approximately 10, 000 cells per second) on a single workstation, which is an encouraging result in whole-slide imaging practice. <s> BIB024 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Glial cells play an important role in regulating synaptogenesis, development of blood-brain barrier, and brain tumor metastasis. Quantitative analysis of glial cells can offer new insights to many studies. However, the complicated morphology of the protrusions of glial cells and the entangled cell-to-cell network cause significant difficulties to extracting quantitative information in images. In this paper, we present a new method for instance-level segmentation of glial cells in 3D images. First, we obtain accurate voxel-level segmentation by leveraging the recent advances of fully convolutional networks (FCN). Then we develop a k-terminal cut algorithm to disentangle the complex cell-to-cell connections. During the cell cutting process, to better capture the nature of glial cells, a shape prior computed based on a multiplicative Voronoi diagram is exploited. Extensive experiments using real 3D images show that our method has superior performance over the state-of-the-art methods. <s> BIB025 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Solid tumors are heterogeneous tissues composed of a mixture of cells and have special tissue architectures. However, cellular heterogeneity, the differences in cell types are generally not reflected in molecular profilers or in recent histopathological image-based analysis of lung cancer, rendering such information underused. This paper presents the development of a computational approach in H&E stained pathological images to quantitatively describe cellular heterogeneity from different types of cells. In our work, a deep learning approach was first used for cell subtype classification. Then we introduced a set of quantitative features to describe cellular information. Several feature selection methods were used to discover significant imaging biomarkers for survival prediction. These discovered imaging biomarkers are consistent with pathological and biological evidence. Experimental results on two lung cancer data sets demonstrated that survival models bsuilt from the clinical imaging biomarkers have better prediction power than state-of-the-art methods using molecular profiling data and traditional imaging biomarkers. <s> BIB026 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Background: Ongoing research into inflammatory conditions raises an increasing need to evaluate immune cells in histological sections in biologically relevant regions of interest (ROIs). Herein, we compare different approaches to automatically detect lobular structures in human normal breast tissue in digitized whole slide images (WSIs). This automation is required to perform objective and consistent quantitative studies on large data sets.Methods: In normal breast tissue from nine healthy patients immunohistochemically stained for different markers, we evaluated and compared three different image analysis methods to automatically detect lobular structures in WSIs: (1) a bottom-up approach using the cell-based data for subsequent tissue level classification, (2) a top-down method starting with texture classification at tissue level analysis of cell densities in specific ROIs, and (3) a direct texture classification using deep learning technology.Results: All three methods result in comparable overall quality allowing automated detection of lobular structures with minor advantage in sensitivity (approach 3), specificity (approach 2), or processing time (approach 1). Combining the outputs of the approaches further improved the precision.Conclusions: Different approaches of automated ROI detection are feasible and should be selected according to the individual needs of biomarker research. Additionally, detected ROIs could be used as a basis for quantification of immune infiltration in lobular structures. HighlightsThe detection of human breast lobular structures in whole slide images can be automated.Three analysis methods (bottom-up, top-down, CNN) are evaluated and compared.All three methods perform well according to quantitative and visual assessments.Precision can be improved by a combination scheme (pixel-level majority voting). <s> BIB027 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Segmentation of 3D images is a fundamental problem in biomedical image analysis. Deep learning (DL) approaches have achieved state-of-the-art segmentation perfor- mance. To exploit the 3D contexts using neural networks, known DL segmentation methods, including 3D convolution, 2D convolution on planes orthogonal to 2D image slices, and LSTM in multiple directions, all suffer incompatibility with the highly anisotropic dimensions in common 3D biomedical images. In this paper, we propose a new DL framework for 3D image segmentation, based on a com- bination of a fully convolutional network (FCN) and a recurrent neural network (RNN), which are responsible for exploiting the intra-slice and inter-slice contexts, respectively. To our best knowledge, this is the first DL framework for 3D image segmentation that explicitly leverages 3D image anisotropism. Evaluating using a dataset from the ISBI Neuronal Structure Segmentation Challenge and in-house image stacks for 3D fungus segmentation, our approach achieves promising results comparing to the known DL-based 3D segmentation approaches. <s> BIB028 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> The morphology of glands has been used routinely by pathologists to assess the malignancy degree of adenocarcinomas. Accurate segmentation of glands from histology images is a crucial step to obtain reliable morphological statistics for quantitative diagnosis. In this paper, we proposed an efficient deep contour-aware network (DCAN) to solve this challenging problem under a unified multi-task learning framework. In the proposed network, multi-level contextual features from the hierarchical architecture are explored with auxiliary supervision for accurate gland segmentation. When incorporated with multi-task regularization during the training, the discriminative capability of intermediate features can be further improved. Moreover, our network can not only output accurate probability maps of glands, but also depict clear contours simultaneously for separating clustered objects, which further boosts the gland segmentation performance. This unified framework can be efficient when applied to large-scale histopathological data without resorting to additional steps to generate contours based on low-level cues for post-separating. Our method won the 2015 MICCAI Gland Segmentation Challenge out of 13 competitive teams, surpassing all the other methods by a significant margin. <s> BIB029 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> In this paper, we study the influence of both long and short skip connections on Fully Convolutional Networks (FCN) for biomedical image segmentation. In standard FCNs, only long skip connections are used to skip features from the contracting path to the expanding path in order to recover spatial information lost during downsampling. We extend FCNs by adding short skip connections, that are similar to the ones introduced in residual networks, in order to build very deep FCNs (of hundreds of layers). A review of the gradient flow confirms that for a very deep FCN it is beneficial to have both long and short skip connections. Finally, we show that a very deep FCN can achieve near-to-state-of-the-art results on the EM dataset without any further post-processing. <s> BIB030 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> We investigate glandular structure segmentation in colon histology images as a window-based classification problem. We compare and combine methods based on fine-tuned convolutional neural networks (CNN) and hand-crafted features with support vector machines (HC-SVM). On 85 images of H&E-stained tissue, we find that fine-tuned CNN outperforms HC-SVM in gland segmentation measured by pixel-wise Jaccard and Dice indices. For HC-SVM we further observe that training a second-level window classifier on the posterior probabilities — as an output refinement — can substantially improve the segmentation performance. The final performance of HC-SVM with refinement is comparable to that of CNN. Furthermore, we show that by combining and refining the posterior probability outputs of CNN and HC-SVM together, a further performance boost is obtained. <s> BIB031 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Recently there has been an increasing trend to use deep learning frameworks for both 2D consumer images and for 3D medical images. However, there has been little effort to use deep frameworks for volumetric vascular segmentation. We wanted to address this by providing a freely available dataset of 12 annotated two-photon vasculature microscopy stacks. We demonstrated the use of deep learning framework consisting both 2D and 3D convolutional filters (ConvNet). Our hybrid 2D-3D architecture produced promising segmentation result. We derived the architectures from Lee et al. who used the ZNN framework initially designed for electron microscope image segmentation. We hope that by sharing our volumetric vasculature datasets, we will inspire other researchers to experiment with vasculature dataset and improve the used network architectures. <s> BIB032 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> To make reliable diagnosis, pathologists often need to identify certain special regions in medical images. In inflammatory bowel disease (IBD) diagnosis via histology tissue image examination, muscle regions are known to have no immune cell infiltration, and thus are ignored by pathologists. Also, messy regions (e.g., due to distortion and poor staining) are low in diagnostic yield. Hence, excluding muscle and messy regions to focus on vital regions is crucial for accurate diagnosis of IBD. In this paper, we propose a novel deep neural network based on fully convolutional networks (FCN) to identify muscle and messy regions, in an end-to-end fashion. First, we address the challenge of having limited medical training data, for training our deep neural network (a common problem for medical image processing, which may impede the application of the powerful deep learning method). Second, to deal with target regions of largely different sizes and arbitrary shapes, our deep neural network explores multi-scale information and structural information. Experimental results on clinical images show that our approach outperforms the state-of-the-art FCN for semantic segmentation of muscle and messy regions. Our approach may be readily extended to identify other types of regions in a variety of medical imaging applications. <s> BIB033 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Accurate segmentation of perimysium plays an important role in early diagnosis of many muscle diseases because many diseases contain different perimysium inflammation. However, it remains as a challenging task due to the complex appearance of the perymisum morphology and its ambiguity to the background area. The muscle perimysium also exhibits strong structure spanned in the entire tissue, which makes it difficult for current local patch-based methods to capture this long-range context information. In this paper, we propose a novel spatial clockwork recurrent neural network (spatial CW-RNN) to address those issues. Specifically, we split the entire image into a set of non-overlapping image patches, and the semantic dependencies among them are modeled by the proposed spatial CW-RNN. Our method directly takes the 2D structure of the image into consideration and is capable of encoding the context information of the entire image into the local representation of each patch. Meanwhile, we leverage on the structured regression to assign one prediction mask rather than a single class label to each local patch, which enables both efficient training and testing. We extensively test our method for perimysium segmentation using digitized muscle microscopy images. Experimental results demonstrate the superiority of the novel spatial CW-RNN over other existing state of the arts. <s> BIB034 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Epithelial (EP) and stromal (ST) are two types of tissues in histological images. Automated segmentation or classification of EP and ST tissues is important when developing computerized system for analyzing the tumor microenvironment. In this paper, a Deep Convolutional Neural Networks (DCNN) based feature learning is presented to automatically segment or classify EP and ST regions from digitized tumor tissue microarrays (TMAs). Current approaches are based on handcraft feature representation, such as color, texture, and Local Binary Patterns (LBP) in classifying two regions. Compared to handcrafted feature based approaches, which involve task dependent representation, DCNN is an end-to-end feature extractor that may be directly learned from the raw pixel intensity value of EP and ST tissues in a data driven fashion. These high-level features contribute to the construction of a supervised classifier for discriminating the two types of tissues. In this work we compare DCNN based models with three handcraft feature extraction based approaches on two different datasets which consist of 157 Hematoxylin and Eosin (H&E) stained images of breast cancer and 1376 immunohistological (IHC) stained images of colorectal cancer, respectively. The DCNN based feature learning approach was shown to have a F1 classification score of 85%, 89%, and 100%, accuracy (ACC) of 84%, 88%, and 100%, and Matthews Correlation Coefficient (MCC) of 86%, 77%, and 100% on two H&E stained (NKI and VGH) and IHC stained data, respectively. Our DNN based approach was shown to outperform three handcraft feature extraction based approaches in terms of the classification of EP and ST regions. <s> BIB035 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Recent advances in computer vision enable increasingly accurate automated pattern classification. In the current study we evaluate whether a convolutional neural network (CNN) can be trained to predict disease outcome in patients with colorectal cancer based on images of tumor tissue microarray samples. We compare the prognostic accuracy of CNN features extracted from the whole, unsegmented tissue microarray spot image, with that of CNN features extracted from the epithelial and non-epithelial compartments, respectively. The prognostic accuracy of visually assessed histologic grade is used as a reference. The image data set consists of digitized hematoxylin-eosin (H and E) stained tissue microarray samples obtained from 180 patients with colorectal cancer. The patient samples represent a variety of histological grades, have data available on a series of clinicopathological variables including long-term outcome and ground truth annotations performed by experts. The CNN features extracted from images of the epithelial tissue compartment significantly predicted outcome (hazard ratio (HR) 2.08; CI95% 1.04-4.16; area under the curve (AUC) 0.66) in a test set of 60 patients, as compared to the CNN features extracted from unsegmented images (HR 1.67; CI95% 0.84-3.31, AUC 0.57) and visually assessed histologic grade (HR 1.96; CI95% 0.99-3.88, AUC 0.61). As a conclusion, a deep-learning classifier can be trained to predict outcome of colorectal cancer based on images of H and E stained tissue microarray samples and the CNN features extracted from the epithelial compartment only resulted in a prognostic discrimination comparable to that of visually determined histologic grade. <s> BIB036 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> We developed an automatic algorithm with the purpose to assist pathologists to report Gleason score on malignant prostatic adenocarcinoma specimen. In order to detect and classify the cancerous tissue, a deep convolutional neural network that had been pre-trained on a large set of photographic images was used. A specific aim was to support intuitive interaction with the result, to let pathologists adjust and correct the output. Therefore, we have designed an algorithm that makes a spatial classification of the whole slide into the same growth patterns as pathologists do. The 22-layer network was cut at an earlier layer and the output from that layer was used to train both a random forest classifier and a support vector machines classifier. At a specific layer a small patch of the image was used to calculate a feature vector and an image is represented by a number of those vectors. We have classified both the individual patches and the entire images. The classification results were compared for different scales of the images and feature vectors from two different layers from the network. Testing was made on a dataset consisting of 213 images, all containing a single class, benign tissue or Gleason score 35. Using 10-fold cross validation the accuracy per patch was 81 %. For whole images, the accuracy was increased to 89 %. <s> BIB037 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Cytopathology is the study of disease at the cellular level and often used as a screening tool for cancer. Thyroid ::: cytopathology is a branch of pathology that studies the diagnosis of thyroid lesions and diseases. A pathologist ::: views cell images that may have high visual variance due to different anatomical structures and pathological ::: characteristics. To assist the physician with identifying and searching through images, we propose a deep semantic ::: mobile application. Our work augments recent advances in the digitization of pathology and machine learning ::: techniques, where there are transformative opportunities for computers to assist pathologists. Our system uses ::: a custom thyroid ontology that can be augmented with multimedia metadata extracted from images using deep ::: machine learning techniques. We describe the utilization of a particular methodology, deep convolutional neural ::: networks, to the application of cytopathology classification. Our method is able to leverage networks that have ::: been trained on millions of generic images, to medical scenarios where only hundreds or thousands of images ::: exist. We demonstrate the benefits of our framework through both quantitative and qualitative results. <s> BIB038 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Pathologists face a substantial increase in workload and complexity of histopathologic cancer diagnosis due to the advent of personalized medicine. Therefore, diagnostic protocols have to focus equally on efficiency and accuracy. In this paper we introduce 'deep learning' as a technique to improve the objectivity and efficiency of histopathologic slide analysis. Through two examples, prostate cancer identification in biopsy specimens and breast cancer metastasis detection in sentinel lymph nodes, we show the potential of this new methodology to reduce the workload for pathologists, while at the same time increasing objectivity of diagnoses. We found that all slides containing prostate cancer and micro- and macro-metastases of breast cancer could be identified automatically while 30-40% of the slides containing benign and normal tissue could be excluded without the use of any additional immunohistochemical markers or human intervention. We conclude that 'deep learning' holds great promise to improve the efficacy of prostate cancer diagnosis and breast cancer staging. <s> BIB039 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Point of care diagnostics using microscopy and computer vision methods have been applied to a number of practical problems, and are particularly relevant to low-income, high disease burden areas. However, this is subject to the limitations in sensitivity and specificity of the computer vision methods used. In general, deep learning has recently revolutionised the field of computer vision, in some cases surpassing human performance for other object recognition tasks. In this paper, we evaluate the performance of deep convolutional neural networks on three different microscopy tasks: diagnosis of malaria in thick blood smears, tuberculosis in sputum samples, and intestinal parasite eggs in stool samples. In all cases accuracy is very high and substantially better than an alternative approach more representative of traditional medical imaging techniques. <s> BIB040 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Early stage estrogen receptor positive (ER+) breast cancer (BCa) treatment is based on the presumed aggressiveness and likelihood of cancer recurrence. The primary conundrum in treatment and management of early stage ER+ BCa is identifying which of these cancers are candidates for adjuvant chemotherapy and which patients will respond to hormonal therapy alone. This decision could spare some patients the inherent toxicity associated with adjuvant chemotherapy. Oncotype DX (ODX) and other gene expression tests have allowed for distinguishing the more aggressive ER+ BCa requiring adjuvant chemotherapy from the less aggressive cancers benefiting from hormonal therapy alone. However these gene expression tests tend to be expensive, tissue destructive and require physical shipping of tissue blocks for the test to be done. Interestingly breast cancer grade in these tumors has been shown to be highly correlated with the ODX risk score. Unfortunately studies have shown that Bloom-Richardson (BR) grade determined by pathologists can be highly variable. One of the constituent categories in BR grading is the quantification of tubules. The goal of this study was to develop a deep learning neural network classifier to automatically identify tubule nuclei from whole slide images (WSI) of ER+ BCa, the hypothesis being that the ratio of tubule nuclei to overall number of nuclei would correlate with the corresponding ODX risk categories. The performance of the tubule nuclei deep learning strategy was evaluated with a set of 61 high power fields. Under a 5-fold cross-validation, the average precision and recall measures were 0:72 and 0:56 respectively. In addition, the correlation with the ODX risk score was assessed in a set of 7513 high power fields extracted from 174 WSI, each from a different patient (At most 50 high power fields per patient study were used). The ratio between the number of tubule and non-tubule nuclei was computed for each WSI. The results suggests that for BCa cases with both low ODX score and low BR grade, the mean tubule nuclei ratio was significantly higher than that obtained for the BCa cases with both high ODX score and high BR grade (p < 0:01). The low ODX and low BR grade cases also presented a significantly higher average tubule nuclei ratio when compared with the rest of the BCa cases (p < 0:05). Finally, the BCa cases that presented both a high ODX and high BR grade show a mean tubule nuclei to total number of nuclei ratio which was significantly smaller than that obtained for the rest of BCa cases (p < 0:01). <s> BIB041 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> The International Symposium on Biomedical Imaging (ISBI) held a grand challenge to evaluate computational systems for the automated detection of metastatic breast cancer in whole slide images of sentinel lymph node biopsies. Our team won both competitions in the grand challenge, obtaining an area under the receiver operating curve (AUC) of 0.925 for the task of whole slide image classification and a score of 0.7051 for the tumor localization task. A pathologist independently reviewed the same images, obtaining a whole slide image classification AUC of 0.966 and a tumor localization score of 0.733. Combining our deep learning system's predictions with the human pathologist's diagnoses increased the pathologist's AUC to 0.995, representing an approximately 85 percent reduction in human error rate. These results demonstrate the power of using deep learning to produce significant improvements in the accuracy of pathological diagnoses. <s> BIB042 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Context: Color normalization techniques for histology have not been empirically tested for their utility for computational pathology pipelines. Aims: We compared two contemporary techniques for achieving a common intermediate goal - epithelial-stromal classification. Settings and Design: Expert-annotated regions of epithelium and stroma were treated as ground truth for comparing classifiers on original and color-normalized images. Materials and Methods: Epithelial and stromal regions were annotated on thirty diverse-appearing H and E stained prostate cancer tissue microarray cores. Corresponding sets of thirty images each were generated using the two color normalization techniques. Color metrics were compared for original and color-normalized images. Separate epithelial-stromal classifiers were trained and compared on test images. Main analyses were conducted using a multiresolution segmentation (MRS) approach; comparative analyses using two other classification approaches (convolutional neural network [CNN], Wndchrm) were also performed. Statistical Analysis: For the main MRS method, which relied on classification of super-pixels, the number of variables used was reduced using backward elimination without compromising accuracy, and test - area under the curves (AUCs) were compared for original and normalized images. For CNN and Wndchrm, pixel classification test-AUCs were compared. Results: Khan method reduced color saturation while Vahadane reduced hue variance. Super-pixel-level test-AUC for MRS was 0.010-0.025 (95% confidence interval limits ± 0.004) higher for the two normalized image sets compared to the original in the 10-80 variable range. Improvement in pixel classification accuracy was also observed for CNN and Wndchrm for color-normalized images. Conclusions: Color normalization can give a small incremental benefit when a super-pixel-based classification method is used with features that perform implicit color normalization while the gain is higher for patch-based classification methods for classifying epithelium versus stroma. <s> BIB043 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Efficient Human Epithelial-2 cell image classification can facilitate the diagnosis of many autoimmune diseases. This paper proposes an automatic framework for this classification task, by utilizing the deep convolutional neural networks (CNNs) which have recently attracted intensive attention in visual recognition. In addition to describing the proposed classification framework, this paper elaborates several interesting observations and findings obtained by our investigation. They include the important factors that impact network design and training, the role of rotation-based data augmentation for cell images, the effectiveness of cell image masks for classification, and the adaptability of the CNN-based classification system across different datasets. Extensive experimental study is conducted to verify the above findings and compares the proposed framework with the well-established image classification models in the literature. The results on benchmark datasets demonstrate that 1) the proposed framework can effectively outperform existing models by properly applying data augmentation, 2) our CNN-based framework has excellent adaptability across different datasets, which is highly desirable for cell image classification under varying laboratory settings. Our system is ranked high in the cell image classification competition hosted by ICPR 2014. <s> BIB044 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> Accurate segmentation of cervical cells in Pap smear images is an important step in automatic pre-cancer identification in the uterine cervix. One of the major segmentation challenges is overlapping of cytoplasm, which has not been well-addressed in previous studies. To tackle the overlapping issue, this paper proposes a learning-based method with robust shape priors to segment individual cell in Pap smear images to support automatic monitoring of changes in cells, which is a vital prerequisite of early detection of cervical cancer. We define this splitting problem as a discrete labeling task for multiple cells with a suitable cost function. The labeling results are then fed into our dynamic multi-template deformation model for further boundary refinement. Multi-scale deep convolutional networks are adopted to learn the diverse cell appearance features. We also incorporated high-level shape information to guide segmentation where cell boundary might be weak or lost due to cell overlapping. An evaluation carried out using two different datasets demonstrates the superiority of our proposed method over the state-of-the-art methods in terms of segmentation accuracy. <s> BIB045 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> A quantitative model to genetically interpret the histology in whole microscopy slide images is desirable to guide downstream immuno-histochemistry, genomics, and precision medicine. We constructed a statistical model that predicts whether or not SPOP is mutated in prostate cancer, given only the digital whole slide after standard hematoxylin and eosin [H&E] staining. Using a TCGA cohort of 177 prostate cancer patients where 20 had mutant SPOP, we trained multiple ensembles of residual networks, accurately distinguishing SPOP mutant from SPOP non-mutant patients. We further validated our full metaensemble classifier on an independent test cohort from MSK-IMPACT of 152 patients where 19 had mutant SPOP. Mutants and non-mutants were accurately distinguished despite TCGA slides being frozen sections and MSK-IMPACT slides being formalin-fixed paraffin-embedded sections. Moreover, we scanned an additional 36 MSK-IMPACT patient having mutant SPOP, trained on this expanded MSK-IMPACT cohort, tested on the TCGA cohort, and again accurately distinguished mutants from non-mutants using the same pipeline. Importantly, our method demonstrates tractable deep learning in this “small data” setting of 20-55 positive examples and quantifies each prediction’s uncertainty with confidence intervals. To our knowledge, this is the first statistical model to predict a genetic mutation in cancer directly from the patient’s digitized H&E-stained whole microscope slide. Significance Statement This is the first automatic pipeline predicting gene mutation probability in cancer from digitized H&E-stained microscopy slides. To predict whether or not the speckle-type POZ protein [SPOP] gene is mutated in prostate cancer, the pipeline (i) identifies diagnostically salient slide regions, (ii) identifies the salient region having the dominant tumor, and (iii) trains ensembles of binary classifiers that together predict a confidence interval of mutation probability. Through deep learning on small datasets, this work enables fully-automated histologic diagnoses based on probabilities of underlying molecular aberrations. <s> BIB046 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> This paper concerns automated cell counting and detection in microscopy images. The approach we take is to use convolutional neural networks (CNNs) to regress a cell spatial density map across the ... <s> BIB047 </s> A Survey on Deep Learning in Medical Image Analysis <s> Chest <s> The capabilities of (I) learning transferable knowledge across domains; and (II) fine-tuning the pre-learned base knowledge towards tasks with considerably smaller data scale are extremely important. Many of the existing transfer learning techniques are supervised approaches, among which deep learning has the demonstrated power of learning domain transferrable knowledge with large scale network trained on massive amounts of labeled data. However, in many biomedical tasks, both the data and the corresponding label can be very limited, where the unsupervised transfer learning capability is urgently needed. In this paper, we proposed a novel multi-scale convolutional sparse coding (MSCSC) method, that (I) automatically learns filter banks at different scales in a joint fashion with enforced scale-specificity of learned patterns; and (II) provides an unsupervised solution for learning transferable base knowledge and fine-tuning it towards target tasks. Extensive experimental evaluation of MSCSC demonstrates the effectiveness of the proposed MSCSC in both regular and transfer learning tasks in various biomedical domains. <s> BIB048
In thoracic image analysis of both radiography and computed tomography, the detection, characterization, and classification of nodules is the most commonly addressed application. Many works add features derived from deep networks to existing feature sets or compare (2013) Detection of basal cell carcinoma H&E Convolutional auto-encoder neural network BIB002 Mitosis detection H&E Combines shapebased features with CNN Wang et al. (2014) Mitosis detection H&E Cascaded ensemble of CNN and handcrafted features BIB004 Bacterial colony counting Culture plate CNN-based patch classifier BIB005 Cell segmentation EM U-Net with deformation augmentation BIB006 Mitosis detection Live-imaging CNN-based patch classifier BIB007 Segmentation of cytoplasm and nuclei H&E Multi-scale CNN and graph-partitioning-based method BIB008 Nucleus detection Ki-67 CNN model that learns the voting offset vectors and voting confidence BIB009 Nucleus detection H&E, Ki-67 CNN-based structured regression model for cell detection BIB011 Cell segmentation FL, PC, H&E fCNN for cell bounding box proposal and CNN for segmentation BIB012 Mitosis detection H&E Incorporated 'crowd sourcing' layer into the CNN framework BIB013 Nucleus classification IHC CNN-based patch classifier BIB014 Mitosis detection H&E Deep regression network (DRN) BIB044 Nucleus classification IFL Classification of Hep2-cells with CNN Han et al. (2016) Nucleus classification IFL Classification of Hep2-cells with CNN Janowczyk et al. (2016b) Nucleus segmentation H&E Resolution adaptive deep hierarchical learning scheme BIB015 Nucleus detection H&E Combination of CNN and hand-crafted features BIB016 Mitosis detection PC Hierarchical CNNs for patch sequence classification BIB017 Classification of mitochondria EM CNN-based patch classifier BIB018 Nucleus classification FL Classification of Hep2-cells using transfer learning (pre-trained CNN) Romo-Bucheli et al. (2016) Tubule nuclei detection H&E CNN-based classification of pre-selected candidate nuclei BIB019 Nucleus detection and classification H&E CNN with spatially constrained regression BIB045 Cell segmentation H&E Multi-scale CNN Turkki et al. (2016) TIL detection H&E CNN-based classification of superpixels BIB020 Nuclear area measurement H&E A CNN directly measures nucleus area without requiring segmentation BIB021 Subtype cell detection H&E Combination of two CNNs for joint cell detection and classification BIB047 Nucleus detection and cell counting FL and H&E Microscopy cell counting with fully convolutional regression networks BIB022 Nucleus segmentation H&E, IHC CNN and selection-based sparse shape model BIB023 Nucleus detection H&E Stacked sparse auto-encoders (SSAE) BIB024 Nucleus detection Various General deep learning framework to detect cells in whole-slide images BIB025 Glial cell segmentation TPM fCNN with an iterative k-terminal cut algorithm BIB026 Nucleus classification H&E Classifies cellular tissue into tumor, lymphocyte, and stromal Classification of leukocytes RM CNN-based patch classifier Large organ segmentation BIB001 Segmentation of neuronal membranes EM Ensemble of several CNNs with different architectures BIB010 Segmentation of colon glands H&E Used two CNNs to segment glands and their separating structures BIB027 Detection of lobular structures in breast IHC Combined the outputs of a CNN and a texture classification system Segmentation of colon glands H&E fCNN with a loss accounting for smoothness and object interactions Segmentation of colon glands H&E A multi-loss fCNN to perform both segmentation and classification BIB028 Neuronal membrane and fungus segmentation EM Combination of bi-directional LSTM-RNNs and kU-Nets BIB029 Segmentation of colon glands H&E Deep contour-aware CNN Ç içek et al. (2016) Segmentation of xenopus kidney CM 3D U-Net BIB030 Segmentation of neuronal structures EM fCNN with skip connections BIB031 Segmentation of colon glands H&E Compares CNN with an SVM using hand-crafted features BIB032 Volumetric vascular segmentation FL Hybrid 2D-3D CNN architecture BIB033 Segmentation of messy and muscle regions H&E Conditional random field jointly trained with an fCNN BIB034 Perimysium segmentation H&E 2D spatial clockwork RNN Xu et al. (2016d) Segmentation of colon glands H&E Used three CNNs to predict gland and contour pixels BIB035 Segmenting epithelium & stroma H&E, IHC CNNs applied to over-segmented image regions (superpixels) Detection and classification of disease Cruz-Roa et al. (2014) Detection of invasive ductal carcinoma H&E CNN-based patch classifier BIB003 Patch-level classification of colon cancer H&E Multiple instance learning framework with CNN features BIB036 Outcome prediction of colorectal cancer H&E Extracted CNN features from epithelial tissue for prediction BIB048 Multiple cancer tissue classification Various Transfer learning using multi-Scale convolutional sparse coding Günhan Ertosun and Rubin (2015) Grading glioma H&E Ensemble of CNNs BIB037 Predicting Gleason score H&E OverFeat pre-trained network as feature extractor BIB038 Thyroid cytopathology classification H&E, RM & Pap Fine-tuning pre-trained AlexNet BIB039 Detection of prostate and breast cancer H&E fCNN-based pixel classifier BIB040 Malaria, tuberculosis and parasites detection Light microscopy CNN-based patch classifier BIB041 Gleason grading and breast cancer detection H&E The system incorporates shearlet features inside a CNN BIB046 SPOP mutation prediction of prostate cancer H&E Ensemble of ResNets BIB042 Metastases BIB043 Comparison of normalization algorithms H&E Presents effectiveness of stain normalization for application of CNNs CNNs with classical machine learning approaches using handcrafted features. In chest X-ray, several groups detect multiple diseases with a single system. In CT the detection of textural patterns indicative of interstitial lung diseases is also a popular research topic. Chest radiography is the most common radiological exam; several works use a large set of images with text reports to train systems that combine CNNs for image analysis and RNNs for text analysis. This is a branch of research we expect to see more of in the near future. In a recent challenge for nodule detection in CT, LUNA16, CNN architectures were used by all top performing systems. This is in contrast with a previous lung nodule detection challenge, ANODE09, where handcrafted features were used to classify nodule candidates. The best systems in LUNA16 still rely on nodule candidates computed by rule-based image processing, but systems that use deep networks for candidate detection also performed very well (e.g. U-net). Estimating the probability that an individual has lung cancer from a CT scan is an important topic: It is the objective of the Kaggle Data Science Bowl 2017, with $1 million in prizes and more than one thousand participating teams.