Generative adversarial networks have been successfully applied to inpainting in natural images. However, the current state-of-the-art models have not yet been widely adopted in the medical imaging domain. In this paper, we investigate the performance of three recently published deep learning based inpainting models: context encoders, semantic image inpainting, and the contextual attention model, applied to chest x-rays, as the chest exam is the most commonly performed radiological procedure. We train these generative models on 1.2M 128 @math 128 patches from 60K healthy x-rays, and learn to predict the center 64 @math 64 region in each patch. We test the models on both the healthy and abnormal radiographs. We evaluate the results by visual inspection and comparing the PSNR scores. The outputs of the models are in most cases highly realistic. We show that the methods have potential to enhance and detect abnormalities. In addition, we perform a 2AFC observer study and show that an experienced human observer performs poorly in detecting inpainted regions, particularly those generated by the contextual attention model.
Several previous works in the medical imaging community have intersections with this work. proposed a method to automatically detect, segment, and remove foreign objects from chest radiographs @cite_21 . They used a kNN classifier to perform pixel level classification and filled the detected regions using texture synthesis, whereas we use deep learning based models which perform better in content restoration. proposed a method to simulate nodules and diffuse infiltrates in chest radiographs @cite_3 . They used projected and CT data to generate patches of nodules and infiltrates, and apply rotation and scaling for post-processing. These patches are then blended in the x-rays. This technique can not generate new normal image structures, such as ribs, and thus cannot be used for inpainting. In terms of using generative models to detect abnormalities in medical images, 's work @cite_1 is similar to ours. However, their paper largely focuses on classification and does not perform inpainting, whereas our work has a focus on the quality of the generated inpainted areas.
@cite_21 @cite_1 @cite_3
2775469102 2016818221 1992382176 98520088
Abstract Machine learning approaches hold great potential for the automated detection of lung nodules on chest radiographs, but training algorithms requires very large amounts of manually annotated radiographs, which are difficult to obtain. The increasing availability of PACS (Picture Archiving and Communication System), is laying the technological foundations needed to make available large volumes of clinical data and images from hospital archives. Binary labels indicating whether a radiograph contains a pulmonary lesion can be extracted at scale, using natural language processing algorithms. In this study, we propose two novel neural networks for the detection of chest radiographs containing pulmonary lesions. Both architectures make use of a large number of weakly-labelled images combined with a smaller number of manually annotated x-rays. The annotated lesions are used during training to deliver a type of visual attention feedback informing the networks about their lesion localisation performance. The first architecture extracts saliency maps from high-level convolutional layers and compares the inferred position of a lesion against the true position when this information is available; a localisation error is then back-propagated along with the softmax classification error. The second approach consists of a recurrent attention model that learns to observe a short sequence of smaller image portions through reinforcement learning; the reward function penalises the exploration of areas, within an image, that are unlikely to contain nodules. Using a repository of over 430,000 historical chest radiographs, we present and discuss the proposed methods over related architectures that use either weakly-labelled or annotated images only. Purpose: Chest radiographs commonly contain projections of foreign objects, such as buttons, brassier clips, jewellery, or pacemakers and wires. The presence of these structures can substantially affect the output of computer analysis of these images. An automated method is presented to detect, segment, and remove foreign objects from chest radiographs.Methods: Detection is performed using supervised pixel classification with a kNN classifier, resulting in a probability estimate per pixel to belong to a projected foreign object. Segmentation is performed by grouping and post-processing pixels with a probability above a certain threshold. Next, the objects are replaced by texture inpainting.Results: The method is evaluated in experiments on 257 chest radiographs. The detection at pixel level is evaluated with receiver operating characteristic analysis on pixels within the unobscured lung fields and an A sub z value of 0.949 is achieved. Free response operator characteristic analysis is performed at the object level, and 95.6 of objects are detected with on average 0.25 false positive detections per image. To investigate the effect of removing the detected objects through inpainting, a texture analysis system for tuberculosis detection is applied to images with and without pathology and with and without foreign object removal. Unprocessed, the texture analysis abnormalitymore » score of normal images with foreign objects is comparable to those with pathology. After removing foreign objects, the texture score of normal images with and without foreign objects is similar, while abnormal images, whether they contain foreign objects or not, achieve on average higher scores.Conclusions: The authors conclude that removal of foreign objects from chest radiographs is feasible and beneficial for automated image analysis.« less Projection images, such as those routinely acquired in radiological practice, are difficult to analyze because multiple 3-D structures superimpose at a single point in the 2-D image. Removal of particular superimposed structures may improve interpretation of these images, both by humans and by computers. This work therefore presents a general method to isolate and suppress structures in 2-D projection images. The focus is on elongated structures, which allows an intensity model of a structure of interest to be extracted using local information only. The model is created from profiles sampled perpendicular to the structure. Profiles containing other structures are detected and removed to reduce the influence on the model. Subspace filtering, using blind source separation techniques, is applied to separate the structure to be suppressed from other structures. By subtracting the modeled structure from the original image a structure suppressed image is created. The method is evaluated in four experiments. In the first experiment ribs are suppressed in 20 artificial radiographs simulated from 3-D lung computed tomography (CT) images. The proposed method with blind source separation and outlier detection shows superior suppression of ribs in simulated radiographs, compared to a simplified approach without these techniques. Additionally, the ability of three observers to discriminate between patches containing ribs and containing no ribs, as measured by the area under the receiver operating characteristic curve (AUC), reduced from 0.99-1.00 on original images to 0.75-0.84 on suppressed images. In the second experiment clavicles are suppressed in 253 chest radiographs. The effect of suppression on clavicle visibility is evaluated using the clavicle contrast and border response, showing a reduction of 78 and 34 , respectively. In the third experiment nodules extracted from CT were simulated close to the clavicles in 100 chest radiographs. It was found that after suppression contrast of the nodules was higher than of the clavicles (1.35 and 0.55, respectively) than on original images (1.83 and 2.46, respectively). In the fourth experiment catheters were suppressed in chest radiographs. The ability of three observers to discriminate between patches originating from 36 images with and 21 images without catheters, as measured by the AUC, reduced from 0.98-0.99 on original images to 0.64-0.74 on suppressed images. We conclude that the presented method can markedly reduce the visibility of elongated structures in chest radiographs and shows potential to enhance diagnosis. Most lung nodules that are missed by radiologists as well as computer-aided detection (CADe) schemes overlap with ribs or clavicles in chest radiographs (CXRs). The purpose of this study was to separate bony structures such as ribs and clavicles from soft tissue in CXRs. To achieve this, we developed anatomically (location-) specific multiple massive-training artificial neural networks (MTANNs) which is a class of pixel-based machine learning combined with total variation (TV) minimization smoothing and a histogram-matching-based consistency processing technique. Multi-resolution MTANNs have previously been developed for rib suppression by use of input CXRs and the corresponding “teaching” images for training. Although they were able to suppress ribs, they did not suppress rib edges, ribs close to the lung wall, and the clavicles very well because the orientation, width, contrast, and density of bones are different from location to location and the capability of a single set of multi-resolution MTANNs is limited. To address this issue, the anatomically specific multiple MTANNs developed in this work were designed to separate bones from soft tissue in different anatomic segments of the lungs. Each of multiple anatomically specific MTANNs was trained with the corresponding anatomic segment in the teaching bone images. The output segmental images from the multiple MTANNs were merged to produce a whole bone image. Total variation minimization smoothing was applied to the bone image for reduction of noise while edges were preserved. This bone image was then subtracted from the original CXR to produce a soft-tissue image where the bones were separated out. In order to ensure the contrast and density in different segments were consistent, a histogram-matching technique was applied to the input segmental images. This new method was compared with the conventional MTANNs by using a database of 110 CXRs with pulmonary nodules. Our new anatomically (location-) specific MTANNs separated rib edges, ribs close to the lung wall, and the clavicles from soft tissue in CXRs to a substantially higher level than the conventional MTANNs did, while the visibility of lung nodules and vessels was maintained. Thus, our image-processing technique for bone-soft-tissue separation by means of our new anatomically specific multiple MTANNs would be potentially useful for radiologists as well as for CAD schemes in detection of lung nodules on CXRs.