A Survey on Deep Learning in Medical Image Analysis <s> Object or lesion classification <s> Cataracts are a clouding of the lens and the leading cause of blindness worldwide. Assessing the presence and severity of cataracts is essential for diagnosis and progression monitoring, as well as to facilitate clinical research and management of the disease. Existing automatic methods for cataract grading utilize a predefined set of image features that may provide an incomplete, redundant, or even noisy representation. In this work, we propose a system to automatically learn features for grading the severity of nuclear cataracts from slit-lamp images. Local filters learned from image patches are fed into a convolutional neural network, followed by a set of recursive neural networks to further extract higher-order features. With these features, support vector regression is applied to determine the cataract grade. The proposed system is validated on a large population-based dataset of \(5378\) images, where it outperforms the state-of-the-art by yielding with respect to clinical grading a mean absolute error (\(\varepsilon \)) of \(0.322\), a \(68.6\,\%\) exact integral agreement ratio (\(R_0\)), a \(86.5\,\%\) decimal grading error \(\le \)0.5 (\(R_{e0.5}\)), and a \(99.1\,\%\) decimal grading error \(\le \)1.0 (\(R_{e1.0}\)). <s> BIB001 </s> A Survey on Deep Learning in Medical Image Analysis <s> Object or lesion classification <s> This paper studies the effectiveness of accomplishing high-level tasks with a minimum of manual annotation and good feature representations for medical images. In medical image analysis, objects like cells are characterized by significant clinical features. Previously developed features like SIFT and HARR are unable to comprehensively represent such objects. Therefore, feature representation is especially important. In this paper, we study automatic extraction of feature representation through deep learning (DNN). Furthermore, detailed annotation of objects is often an ambiguous and challenging task. We use multiple instance learning (MIL) framework in classification training with deep learning features. Several interesting conclusions can be drawn from our work: (1) automatic feature learning outperforms manual feature; (2) the unsupervised approach can achieve performance that's close to fully supervised approach (93.56%) vs. (94.52%); and (3) the MIL performance of coarse label (96.30%) outweighs the supervised performance of fine label (95.40%) in supervised deep learning features. <s> BIB002 </s> A Survey on Deep Learning in Medical Image Analysis <s> Object or lesion classification <s> We investigate the problem of diagnostic lung nodule classification using thoracic Computed Tomography (CT) screening. Unlike traditional studies primarily relying on nodule segmentation for regional analysis, we tackle a more challenging problem on directly modelling raw nodule patches without any prior definition of nodule morphology. We propose a hierarchical learning framework—Multi-scale Convolutional Neural Networks (MCNN)—to capture nodule heterogeneity by extracting discriminative features from alternatingly stacked layers. In particular, to sufficiently quantify nodule characteristics, our framework utilizes multi-scale nodule patches to learn a set of class-specific features simultaneously by concatenating response neuron activations obtained at the last layer from each input scale. We evaluate the proposed method on CT images from Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI), where both lung nodule screening and nodule annotations are provided. Experimental results demonstrate the effectiveness of our method on classifying malignant and benign nodules without nodule segmentation. <s> BIB003 </s> A Survey on Deep Learning in Medical Image Analysis <s> Object or lesion classification <s> Correctly classifying a skin lesion is one of the first steps towards treatment. We propose a novel convolutional neural network (CNN) architecture for skin lesion classification designed to learn based on information from multiple image resolutions while leveraging pretrained CNNs. While traditional CNNs are generally trained on a single resolution image, our CNN is composed of multiple tracts, where each tract analyzes the image at a different resolution simultaneously and learns interactions across multiple image resolutions using the same field-of-view. We convert a CNN, pretrained on a single resolution, to work for multi-resolution input. The entire network is fine-tuned in a fully learned end-to-end optimization with auxiliary loss functions. We show how our proposed novel multi-tract network yields higher classification accuracy, outperforming state-of-the-art multi-scale approaches when compared over a public skin lesion dataset. <s> BIB004 </s> A Survey on Deep Learning in Medical Image Analysis <s> Object or lesion classification <s> We propose a novel Computer-Aided Detection (CAD) system for pulmonary nodules using multi-view convolutional networks (ConvNets), for which discriminative features are automatically learnt from the training data. The network is fed with nodule candidates obtained by combining three candidate detectors specifically designed for solid, subsolid, and large nodules. For each candidate, a set of 2-D patches from differently oriented planes is extracted. The proposed architecture comprises multiple streams of 2-D ConvNets, for which the outputs are combined using a dedicated fusion method to get the final classification. Data augmentation and dropout are applied to avoid overfitting. On 888 scans of the publicly available LIDC-IDRI dataset, our method reaches high detection sensitivities of 85.4% and 90.1% at 1 and 4 false positives per scan, respectively. An additional evaluation on independent datasets from the ANODE09 challenge and DLCST is performed. We showed that the proposed multi-view ConvNets is highly suited to be used for false positive reduction of a CAD system. <s> BIB005 </s> A Survey on Deep Learning in Medical Image Analysis <s> Object or lesion classification <s> High-grade glioma is the most aggressive and severe brain tumor that leads to death of almost 50 % patients in 1–2 years. Thus, accurate prognosis for glioma patients would provide essential guidelines for their treatment planning. Conventional survival prediction generally utilizes clinical information and limited handcrafted features from magnetic resonance images (MRI), which is often time consuming, laborious and subjective. In this paper, we propose using deep learning frameworks to automatically extract features from multi-modal preoperative brain images (i.e., T1 MRI, fMRI and DTI) of high-grade glioma patients. Specifically, we adopt 3D convolutional neural networks (CNNs) and also propose a new network architecture for using multi-channel data and learning supervised features. Along with the pivotal clinical features, we finally train a support vector machine to predict if the patient has a long or short overall survival (OS) time. Experimental results demonstrate that our methods can achieve an accuracy as high as 89.9 % We also find that the learned features from fMRI and DTI play more important roles in accurately predicting the OS time, which provides valuable insights into functional neuro-oncological applications. <s> BIB006 </s> A Survey on Deep Learning in Medical Image Analysis <s> Object or lesion classification <s> The choice of features greatly influences the performance of a tissue classification system. Despite this, many systems are built with standard, predefined filter banks that are not optimized for that particular application. Representation learning methods such as restricted Boltzmann machines may outperform these standard filter banks because they learn a feature description directly from the training data. Like many other representation learning methods, restricted Boltzmann machines are unsupervised and are trained with a generative learning objective; this allows them to learn representations from unlabeled data, but does not necessarily produce features that are optimal for classification. In this paper we propose the convolutional classification restricted Boltzmann machine, which combines a generative and a discriminative learning objective. This allows it to learn filters that are good both for describing the training data and for classification. We present experiments with feature learning for lung texture classification and airway detection in CT images. In both applications, a combination of learning objectives outperformed purely discriminative or generative learning, increasing, for instance, the lung tissue classification accuracy by 1 to 8 percentage points. This shows that discriminative learning can help an otherwise unsupervised feature learner to learn filters that are optimized for classification. <s> BIB007 </s> A Survey on Deep Learning in Medical Image Analysis <s> Object or lesion classification <s> Abstract This study aims to build a deep learning (DL) architecture for automated extraction of learned-from-data image features from the shear-wave elastography (SWE), and to evaluate the DL architecture in differentiation between benign and malignant breast tumors. We construct a two-layer DL architecture for SWE feature extraction, comprised of the point-wise gated Boltzmann machine (PGBM) and the restricted Boltzmann machine (RBM). The PGBM contains task-relevant and task-irrelevant hidden units, and the task-relevant units are connected to the RBM. Experimental evaluation was performed with five-fold cross validation on a set of 227 SWE images, 135 of benign tumors and 92 of malignant tumors, from 121 patients. The features learned with our DL architecture were compared with the statistical features quantifying image intensity and texture. Results showed that the DL features achieved better classification performance with an accuracy of 93.4%, a sensitivity of 88.6%, a specificity of 97.1%, and an area under the receiver operating characteristic curve of 0.947. The DL-based method integrates feature learning with feature selection on SWE. It may be potentially used in clinical computer-aided diagnosis of breast cancer. <s> BIB008 </s> A Survey on Deep Learning in Medical Image Analysis <s> Object or lesion classification <s> Mammographic risk scoring has commonly been automated by extracting a set of handcrafted features from mammograms, and relating the responses directly or indirectly to breast cancer risk. We present a method that learns a feature hierarchy from unlabeled data. When the learned features are used as the input to a simple classifier, two different tasks can be addressed: i) breast density segmentation, and ii) scoring of mammographic texture. The proposed model learns features at multiple scales. To control the models capacity a novel sparsity regularizer is introduced that incorporates both lifetime and population sparsity. We evaluated our method on three different clinical datasets. Our state-of-the-art results show that the learned breast density scores have a very strong positive relationship with manual ones, and that the learned texture scores are predictive of breast cancer. The model is easy to apply and generalizes to many other segmentation and scoring problems. <s> BIB009
Object classification usually focuses on the classification of a small (previously identified) part of the medical image into two or more classes (e.g. nodule classification in chest CT). For many of these tasks both local information on lesion appearance and global contextual information on lesion location are required for accurate classification. This combination is typically not possible in generic deep learning architectures. Several authors have used multi-stream architectures to resolve this in a multi-scale fashion (Section 2.4.2). BIB003 used three CNNs, each of which takes a nodule patch at a different scale as input. The resulting feature outputs of the three CNNs are then concatenated to form the final feature vector. A somewhat similar approach was followed by BIB004 who used a multi-stream CNN to classify skin lesions, where each stream works on a different resolution of the image. BIB001 proposed to use a combination of CNNs and RNNs for grading nuclear cataracts in slit-lamp images, where CNN filters were pre-trained. This combination allows the processing of all contextual information regardless of image size. Incorporating 3D information is also often a necessity for good performance in object classification tasks in medical imaging. As images in computer vision tend to be 2D natural images, networks developed in those scenarios do not directly leverage 3D information. Authors have used different approaches to integrate 3D in an effective manner with custom architectures. BIB005 used a multi-stream CNN to classify points of interest in chest CT as a nodule or non-nodule. Up to nine differently oriented patches extracted from the candidate were used in separate streams and merged in the fully-connected layers to obtain the final classification output. In contrast, BIB006 exploited the 3D nature of MRI by training a 3D CNN to assess survival in patients suffering from high-grade gliomas. Almost all recent papers prefer the use of end-to-end trained CNNs. In some cases other architectures and approaches are used, such as RBMs BIB007 BIB008 , SAEs  and convolutional sparse auto-encoders (CSAE) BIB009 . The major difference between CSAE and a classic CNN is the usage of unsupervised pre-training with sparse auto-encoders. An interesting approach, especially in cases where object annotation to generate training data is expensive, is the integration of multiple instance learning (MIL) and deep learning. BIB002 investigated the use of a MIL-framework with both supervised and unsu-pervised feature learning approaches as well as handcrafted features. The results demonstrated that the performance of the MIL-framework was superior to handcrafted features, which in turn closely approaches the performance of a fully supervised method. We expect such approaches to be popular in the future as well, as obtaining high-quality annotated medical data is challenging. Overall, object classification sees less use of pretrained networks compared to exam classifications, mostly due to the need for incorporation of contextual or three-dimensional information. Several authors have found innovative solutions to add this information to deep networks with good results, and as such we expect deep learning to become even more prominent for this task in the near future.