Deep learning algorithms offer a powerful means to automatically analyze the content of medical images. However, many biological samples of interest are primarily transparent to visible light and contain features that are difficult to resolve with a standard optical microscope. Here, we use a convolutional neural network (CNN) not only to classify images, but also to optimize the physical layout of the imaging device itself. We increase the classification accuracy of a microscope's recorded images by merging an optical model of image formation into the pipeline of a CNN. The resulting network simultaneously determines an ideal illumination arrangement to highlight important sample features during image acquisition, along with a set of convolutional weights to classify the detected images post-capture. We demonstrate our joint optimization technique with an experimental microscope configuration that automatically identifies malaria-infected cells with 5-10 higher accuracy than standard and alternative microscope lighting designs.
Deep learning networks for conventional @cite_4 and biomedical @cite_1 image classification are now commonplace and much of this recent work relies on CNNs. However, as noted above, most studies in this area do not try to use deep learning to optimize the acquisition process for their image data sets. While early work has shown that simple neural networks offer an effective way to design cameras @cite_17 , one of the first works to consider this question in the context of CNN-based learning was presented recently by Chakrabarti @cite_5 , who designed an optimal pixel-level color filter layout for color image reconstruction that outperformed the standard Bayer filter pattern.
@cite_1 @cite_5 @cite_4 @cite_17
2592929672 2404325329 2009950757
Abstract Deep learning algorithms, in particular convolutional networks, have rapidly become a methodology of choice for analyzing medical images. This paper reviews the major deep learning concepts pertinent to medical image analysis and summarizes over 300 contributions to the field, most of which appeared in the last year. We survey the use of deep learning for image classification, object detection, segmentation, registration, and other tasks. Concise overviews are provided of studies per application area: neuro, retinal, pulmonary, digital pathology, breast, cardiac, abdominal, musculoskelet al. We end with a summary of the current state-of-the-art, a critical discussion of open challenges and directions for future research. Recent progress on many imaging and vision tasks has been driven by the use of deep feed-forward neural networks, which are trained by propagating gradients of a loss defined on the final output, back through the network up to the first layer that operates directly on the image. We propose back-propagating one step further---to learn camera sensor designs jointly with networks that carry out inference on the images they capture. In this paper, we specifically consider the design and inference problems in a typical color camera---where the sensor is able to measure only one color channel at each pixel location, and computational inference is required to reconstruct a full color image. We learn the camera sensor's color multiplexing pattern by encoding it as layer whose learnable weights determine which color channel, from among a fixed set, will be measured at each location. These weights are jointly trained with those of a reconstruction network that operates on the corresponding sensor measurements to produce a full color image. Our network achieves significant improvements in accuracy over the traditional Bayer pattern used in most color cameras. It automatically learns to employ a sparse color measurement approach similar to that of a recent design, and moreover, improves upon that design by learning an optimal layout for these measurements. The graded-response Hopfield neural network model has been used to solve the traveling salesman optimization problem. However, the mapping of an optical design optimization problem onto a neural net is more difficult. This paper describes how it can be done for the case of minimizing the chromatic aberration in a complicated twenty-element zoom-lens system by the selection of glass types. The problem is combinatorial in nature. It is suited to neural networks, and its solution is non-trivial by other means. Thus the use of neural networks to solve optical optimization problems is demonstrated.© (1993) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.