A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> Image representation plays an important role in medical image analysis. The key to the success of different medical image analysis algorithms is heavily dependent on how we represent the input data, namely features used to characterize the input image. In the literature, feature engineering remains as an active research topic, and many novel hand-crafted features are designed such as Haar wavelet, histogram of oriented gradient, and local binary patterns. However, such features are not designed with the guidance of the underlying dataset at hand. To this end, we argue that the most effective features should be designed in a learning based manner, namely representation learning, which can be adapted to different patient datasets at hand. In this paper, we introduce a deep learning framework to achieve this goal. Specifically, a stacked independent subspace analysis (ISA) network is adopted to learn the most effective features in a hierarchical and unsupervised manner. The learnt features are adapted to the dataset at hand and encode high level semantic anatomical information. The proposed method is evaluated on the application of automatic prostate MR segmentation. Experimental results show that significant segmentation accuracy improvement can be achieved by the proposed deep learning method compared to other state-of-the-art segmentation approaches. <s> BIB001 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> Accurate localization and identification of vertebrae in 3D spinal images is essential for many clinical tasks. However, automatic localization and identification of vertebrae remains challenging due to similar appearance of vertebrae, abnormal pathological curvatures and image artifacts induced by surgical implants. Traditional methods relying on hand-crafted low level features and/or a priori knowledge usually fail to overcome these challenges on arbitrary CT scans. We present a robust and efficient approach to automatically locating and identifying vertebrae in 3D CT volumes by exploiting high level feature representations with deep convolutional neural network CNN. A novel joint learning model with CNN J-CNN is proposed by considering both the appearance of vertebrae and the pairwise conditional dependency of neighboring vertebrae. The J-CNN can effectively identify the type of vertebra and eliminate false detections based on a set of coarse vertebral centroids generated from a random forest classifier. Furthermore, the predicted centroids are refined by a shape regression model. Our approach was quantitatively evaluated on the dataset of MICCAI 2014 Computational Challenge on Vertebrae Localization and Identification. Compared with the state-of-the-art methodi¾?[1], our approach achieved a large margin with 10.12% improvement of the identification rate and smaller localization errors. <s> BIB002 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> Localization of the vertebrae is of importance in many medical applications. For example, the vertebrae can serve as the landmarks in image registration. They can also provide a reference coordinate system to facilitate the localization of other organs in the chest. In this paper, we propose a new vertebrae localization method using convolutional neural networks (CNN). The main advantage of the proposed method is the removal of hand-crafted features. We construct two training sets to train two CNNs that share the same architecture. One is used to distinguish the vertebrae from other tissues in the chest, and the other is aimed at detecting the centers of the vertebrae. The architecture contains two convolutional layers, both of which are followed by a max-pooling layer. Then the output feature vector from the maxpooling layer is fed into a multilayer perceptron (MLP) classifier which has one hidden layer. Experiments were performed on ten chest CT images. We used leave-one-out strategy to train and test the proposed method. Quantitative comparison between the predict centers and ground truth shows that our convolutional neural networks can achieve promising localization accuracy without hand-crafted features. <s> BIB003 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> This paper proposes an automatic method for vertebra localization, labeling, and segmentation in multi-slice Magnetic Resonance (MR) images. Prior work in this area on MR images mostly requires user interaction while our method is fully automatic. Cubic intensity-based features are extracted from image voxels. A deep learning approach is used for simultaneous localization and identification of vertebrae. The localized points are refined by local thresholding in the region of the detected vertebral column. Thereafter, a statistical multi-vertebrae model is initialized on the localized vertebrae. An iterative Expectation Maximization technique is used to register the vertebral body of the model to the image edges and obtain a segmentation of the lumbar vertebral bodies. The method is evaluated by applying to nine volumetric MR images of the spine. The results demonstrate 100% vertebra identification and a mean surface error of below 2.8 mm for 3D segmentation. Computation time is less than three minutes per high-resolution volumetric image. <s> BIB004 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> Accurate localization of the anatomical landmarks on distal femur bone in the 3D medical images is very important for knee surgery planning and biomechanics analysis. However, the landmark identification process is often conducted manually or by using the inserted auxiliaries, which is time-consuming and lacks of accuracy. In this paper, an automatic localization method is proposed to determine positions of initial geometric landmarks on femur surface in the 3D MR images. Based on the results from the convolutional neural network (CNN) classifiers and shape statistics, we use the narrow-band graph cut optimization to achieve the 3D segmentation of femur surface. Finally, the anatomical landmarks are located on the femur according to the geometric cues of surface mesh. Experiments demonstrate that the proposed method is effective, efficient, and reliable to segment femur and locate the anatomical landmarks. <s> BIB005 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> Prostate segmentation on 3D MR images is a challenging task due to image artifacts, large inter-patient prostate shape and texture variability, and lack of a clear prostate boundary specifically at apex and base levels. We propose a supervised machine learning model that combines atlas based Active Appearance Model (AAM) with a Deep Learning model to segment the prostate on MR images. The performance of the segmentation method is evaluated on 20 unseen MR image datasets. The proposed method combining AAM and Deep Learning achieves a mean Dice Similarity Coefficient (DSC) of 0.925 for whole 3D MR images of the prostate using axial cross-sections. The proposed model utilizes the adaptive atlas-based AAM model and Deep Learning to achieve significant segmentation accuracy. <s> BIB006 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> Automatic and reliable segmentation of the prostate is an important but difficult task for various clinical applications such as prostate cancer radiotherapy. The main challenges for accurate MR prostate localization lie in two aspects: (1) inhomogeneous and inconsistent appearance around prostate boundary, and (2) the large shape variation across different patients. To tackle these two problems, we propose a new deformable MR prostate segmentation method by unifying deep feature learning with the sparse patch matching. First, instead of directly using handcrafted features, we propose to learn the latent feature representation from prostate MR images by the stacked sparse auto-encoder (SSAE). Since the deep learning algorithm learns the feature hierarchy from the data, the learned features are often more concise and effective than the handcrafted features in describing the underlying data. To improve the discriminability of learned features, we further refine the feature representation in a supervised fashion. Second, based on the learned features, a sparse patch matching method is proposed to infer a prostate likelihood map by transferring the prostate labels from multiple atlases to the new prostate MR image. Finally, a deformable segmentation is used to integrate a sparse shape model with the prostate likelihood map for achieving the final segmentation. The proposed method has been extensively evaluated on the dataset that contains 66 T2-wighted prostate MR images. Experimental results show that the deep-learned features are more effective than the handcrafted features in guiding MR prostate segmentation. Moreover, our method shows superior performance than other state-of-the-art segmentation methods. <s> BIB007 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> Convolutional Neural Networks (CNNs) have been recently employed to solve problems from both the computer vision and medical image analysis fields. Despite their popularity, most approaches are only able to process 2D images while most medical data used in clinical practice consists of 3D volumes. In this work we propose an approach to 3D image segmentation based on a volumetric, fully convolutional, neural network. Our CNN is trained end-to-end on MRI volumes depicting prostate, and learns to predict segmentation for the whole volume at once. We introduce a novel objective function, that we optimise during training, based on Dice coefficient. In this way we can deal with situations where there is a strong imbalance between the number of foreground and background voxels. To cope with the limited number of annotated volumes available for training, we augment the data applying random non-linear transformations and histogram matching. We show in our experimental evaluation that our approach achieves good performances on challenging test data while requiring only a fraction of the processing time needed by other previous methods. <s> BIB008 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> This paper proposes a new approach to automatically quantify the severity of knee osteoarthritis (OA) from radiographs using deep convolutional neural networks (CNN). Clinically, knee OA severity is assessed using Kellgren \& Lawrence (KL) grades, a five point scale. Previous work on automatically predicting KL grades from radiograph images were based on training shallow classifiers using a variety of hand engineered features. We demonstrate that classification accuracy can be significantly improved using deep convolutional neural network models pre-trained on ImageNet and fine-tuned on knee OA images. Furthermore, we argue that it is more appropriate to assess the accuracy of automatic knee OA severity predictions using a continuous distance-based evaluation metric like mean squared error than it is to use classification accuracy. This leads to the formulation of the prediction of KL grades as a regression problem and further improves accuracy. Results on a dataset of X-ray images and KL grades from the Osteoarthritis Initiative (OAI) show a sizable improvement over the current state-of-the-art. <s> BIB009 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> Abstract Automatic vertebra recognition, including the identification of vertebra locations and naming in multiple image modalities, are highly demanded in spinal clinical diagnoses where large amount of imaging data from various of modalities are frequently and interchangeably used. However, the recognition is challenging due to the variations of MR/CT appearances or shape/pose of the vertebrae. In this paper, we propose a method for multi-modal vertebra recognition using a novel deep learning architecture called Transformed Deep Convolution Network (TDCN). This new architecture can unsupervisely fuse image features from different modalities and automatically rectify the pose of vertebra. The fusion of MR and CT image features improves the discriminativity of feature representation and enhances the invariance of the vertebra pattern, which allows us to automatically process images from different contrast, resolution, protocols, even with different sizes and orientations. The feature fusion and pose rectification are naturally incorporated in a multi-layer deep learning network. Experiment results show that our method outperforms existing detection methods and provides a fully automatic location+naming+pose recognition for routine clinical practice. <s> BIB010 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> Developmental dysplasia of the hip (DDH) is a condition affecting up to 1 in 30 infants. DDH is easy to treat if diagnosed early, but undiagnosed DDH can result in life-long hip pain, dysfunction and an increased risk of early onset osteoarthritis, and accounts for around 30 % of all hip replacements in patients under 60. The gold standard for diagnosis in infants is an ultrasound scan, followed by an analysis procedure known as Graf’s method. The application of Graf’s method is notoriously operator-dependent, requiring years of training to reach reasonable and reproducible performance. We describe a novel deep-learning based pipeline that applies Graf’s method to ultrasound scans of the hip. We use a convolutional network with an adversarial component to segment the image into relevant landmarks, and define a set of post-processing rules to translate the segmentations into Graf’s metrics. Comparing our pipeline to estimates made by experts in DDH diagnosis shows promising results. <s> BIB011 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> We propose an automated method for supervised segmentation of vertebral bodies (VBs) from three-dimensional (3D) magnetic resonance (MR) spine images that is based on coupling deformable models with convolutional neural networks (CNNs). We designed a 3D CNN architecture that learns the appearance from a training set of VBs to generate 3D spatial VB probability maps, which guide deformable models towards VB boundaries. The proposed method was applied to segment 161 VBs from 3D MR spine images of 23 subjects, and the results were compared to reference segmentations. By yielding an overall Dice similarity coefficient of \(93.4\,{\pm }\,1.7\,\%\), mean symmetric surface distance of \(0.54\,{\pm }\,0.14\,\text {mm}\) and Hausdorff distance of \(3.83\,{\pm }\,1.04\,\text {mm}\), the proposed method proved superior to existing VB segmentation methods. <s> BIB012 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> We describe a method to automatically predict radiological scores in spinal Magnetic Resonance Images (MRIs). Furthermore, we also identify and localize the pathologies that are the reasons for these scores. We term these pathological regions the “evidence hotspots”. Our contributions are two-fold: (i) a Convolutional Neural Network (CNN) architecture and training scheme to predict multiple radiological scores on multiple slice sagittal MRIs. The scheme uses multi-task CNN training with augmentation, and handles the class imbalance common in medical classification tasks. (ii) the prediction of a heat-map of evidence hotspots for each score. For both of these, all that is required for training is the class label of the disc or vertebrae, no stronger supervision (such as slice labels) is needed. We report state-of-the-art and near-human performances across multiple radiological scorings including: Pfirrmann grading, disc narrowing, endplate defects, and marrow changes. <s> BIB013 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> In this paper, we present a Convolutional Neural Network (CNN) regression approach to address the two major limitations of existing intensity-based 2-D/3-D registration technology: 1) slow computation and 2) small capture range. Different from optimization-based methods, which iteratively optimize the transformation parameters over a scalar-valued metric function representing the quality of the registration, the proposed method exploits the information embedded in the appearances of the digitally reconstructed radiograph and X-ray images, and employs CNN regressors to directly estimate the transformation parameters. An automatic feature extraction step is introduced to calculate 3-D pose-indexed features that are sensitive to the variables to be regressed while robust to other factors. The CNN regressors are then trained for local zones and applied in a hierarchical manner to break down the complex regression task into multiple simpler sub-tasks that can be learned separately. Weight sharing is furthermore employed in the CNN regression model to reduce the memory footprint. The proposed approach has been quantitatively evaluated on 3 potential clinical applications, demonstrating its significant advantage in providing highly accurate real-time 2-D/3-D registration with a significantly enlarged capture range when compared to intensity-based methods. <s> BIB014 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> Biological age (BA) estimation from radiologic data is an important topic in clinical medicine, e.g. in determining endocrinological diseases or planning paediatric orthopaedic surgeries, while in legal medicine it is employed to approximate chronological age. In this work, we propose the use of deep convolutional neural networks (DCNN) for automatic BA estimation from hand MRI volumes, inspired by the way radiologists visually perform age estimation using established staging schemes that follow physical maturation. In our results we outperform the state of the art automatic BA estimation method, achieving a mean error between estimated and ground truth BA of \(0.36\,\pm \,0.30\) years, which is in line with radiologists doing visual BA estimation. <s> BIB015 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> Colorectal cancer (CRC) is a leading cause of cancer deaths worldwide. Although polypectomy at early stage reduces CRC incidence, 90% of the polyps are small and diminutive, where removal of them poses risks to patients that may outweigh the benefits. Correctly detecting and predicting polyp type during colonoscopy allows endoscopists to resect and discard the tissue without submitting it for histology, saving time, and costs. Nevertheless, human visual observation of early stage polyps varies. Therefore, this paper aims at developing a fully automatic algorithm to detect and classify hyperplastic and adenomatous colorectal polyps. Adenomatous polyps should be removed, whereas distal diminutive hyperplastic polyps are considered clinically insignificant and may be left in situ . A novel transfer learning application is proposed utilizing features learned from big nonmedical datasets with 1.4-2.5 million images using deep convolutional neural network. The endoscopic images we collected for experiment were taken under random lighting conditions, zooming and optical magnification, including 1104 endoscopic nonpolyp images taken under both white-light and narrowband imaging (NBI) endoscopy and 826 NBI endoscopic polyp images, of which 263 images were hyperplasia and 563 were adenoma as confirmed by histology. The proposed method identified polyp images from nonpolyp images in the beginning followed by predicting the polyp histology. When compared with visual inspection by endoscopists, the results of this study show that the proposed method has similar precision (87.3% versus 86.4%) but a higher recall rate (87.6% versus 77.0%) and a higher accuracy (85.9% versus 74.3%). In conclusion, automatic algorithms can assist endoscopists in identifying polyps that are adenomatous but have been incorrectly judged as hyperplasia and, therefore, enable timely resection of these polyps at an early stage before they develop into invasive cancer. <s> BIB016 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> The purpose of this study was to investigate the potential of using clinically provided spine label annotations stored in a single institution image archive as training data for deep learning-based vertebral detection and labeling pipelines. Lumbar and cervical magnetic resonance imaging cases with annotated spine labels were identified and exported from an image archive. Two separate pipelines were configured and trained for lumbar and cervical cases respectively, using the same setup with convolutional neural networks for detection and parts-based graphical models to label the vertebrae. The detection sensitivity, precision and accuracy rates ranged between 99.1–99.8, 99.6–100, and 98.8–99.8% respectively, the average localization error ranges were 1.18–1.24 and 2.38–2.60 mm for cervical and lumbar cases respectively, and with a labeling accuracy of 96.0–97.0%. Failed labeling results typically involved failed S1 detections or missed vertebrae that were not fully visible on the image. These results show that clinically annotated image data from one image archive is sufficient to train a deep learning-based pipeline for accurate detection and labeling of MR images depicting the spine. Further, these results support using deep learning to assist radiologists in their work by providing highly accurate labels that only require rapid confirmation. <s> BIB017 </s> A Survey on Deep Learning in Medical Image Analysis <s> Musculoskeletal <s> Abstract Skeletal bone age assessment is a common clinical practice to investigate endocrinology, genetic and growth disorders in children. It is generally performed by radiological examination of the left hand by using either the Greulich and Pyle (GP nevertheless, none of them has been proved able to generalize to different races, age ranges and genders. In this paper, we propose and test several deep learning approaches to assess skeletal bone age automatically; the results showed an average discrepancy between manual and automatic evaluation of about 0.8 years, which is state-of-the-art performance. Furthermore, this is the first automated skeletal bone age assessment work tested on a public dataset and for all age ranges, races and genders, for which the source code is available, thus representing an exhaustive baseline for future research in the field. Beside the specific application scenario, this paper aims at providing answers to more general questions about deep learning on medical images: from the comparison between deep-learned features and manually-crafted ones, to the usage of deep-learning methods trained on general imagery for medical problems, to how to train a CNN with few images. <s> BIB018
Musculoskeletal images have also been analyzed by deep learning algorithms for segmentation and identification of bone, joint, and associated soft tissue abnormalities in diverse imaging modalities. The works are summarized in Table 9 . Colitis detection CT CNN Pre-trained ImageNet CNN generates features for linear SVM Nappi et al. (2016) Polyp detection CT CNN Substantial reduction of false positives using pre-trained and fine-tuned CNN Tachibana et al. (2016) Electronic cleansing CT CNN Voxel classification in dual energy CT, material other than soft tissue is removed BIB016 Polyp detection Colonoscopy CNN Pre-trained ImageNet CNN for feature extraction, two SVMs for cascaded classification Prostate segmentation in MRI BIB001 Application of stacked independent subspace analysis networks BIB006 CNN produces energy map for 2D slice based active appearance segmentation BIB007 Stacked sparse auto-encoders extract features from patches, input to atlas matching and a deformable model BIB008 3D U-net based CNN architecture with objective function that directly optimizes Dice coefficient, ranks #5 in PROMISE12  3D fully convolutional network, hybrid between a ResNet and U-net architecture, ranks #1 on PROMISE12 BIB002 CT Vertebrae localization; joint learning of vertebrae appearance and dependency on neighbors using CNN Roth et al. (2015c) CT Sclerotic metastases detection; random 2D views are analyzed by CNN and aggregated BIB003 CT Vertebrae localization and segmentation; CNN for segmenting vertebrae and for center detection BIB004 MRI Vertebrae localization, identification and segmentation of vertebrae; CNN used for initial localization BIB005 MRI Anatomical landmark detection; uses CNN for slice classification for presence of landmark BIB009 X-ray Osteoarthritis grading; pre-trained ImageNet CNN fine-tuned on knee X-rays BIB010 CT, MRI Vertebrae localization; RBM determines position, orientation and label of vertebrae BIB011 US Hip dysplasia detection; CNN with adversarial component detects structures and performs measurements BIB012 MRI Vertebral bodies segmentation; voxel probabilities obtained with a 3D CNN are input to deformable model BIB013 MRI Automatic spine scoring; VGG-19 CNN analyzes vertebral discs and finds lesion hotspots BIB014 X-ray Total Knee Arthroplasty kinematics by real-time 2D/3D registration using CNN Roth et al. (2016c) CT Posterior-element fractures detection; CNN for 2.5D patch-based analysiš BIB015 MRI Hand age estimation; 2D regression CNN analyzes 13 bones BIB017 MRI Vertebrae detection and labeling; outputs of two CNNs are input to graphical model BIB018 X-ray Skeletal bone age assessment; comparison among several deep learning approaches for the task at hand A surprising number of complete applications with promising results are available; one that stands out is BIB013 who trained their system with 12K discs and claimed near-human performances across four different radiological scoring tasks.