TECHNICAL FIELD

In at least one aspect, the present invention is related medical imaging techniques.

BACKGROUND

Dielectric images, which provide the permittivity and conductivity information of an object, have shown great potentials in a wide range of biomedical applications, such as medical diagnostics, thermal therapy monitoring and bioelectronics studies, etc. [1]. Dielectric images can be acquired noninvasively through microwave inverse scattering imaging. However, due to the ill-posedness, nonlinearity and non-uniqueness of microwave imaging problem [2], current microwave imaging methods have not yet generated dielectric images of sufficient quality for broad clinical use.

To address these challenges, physical-model-based optimization methods [3] [4] [5] and artificial neural-network-based learning methods [6] [7] have been proposed and investigated for microwave imaging. For the physical-model-based iterative optimization methods, the fields (or contrast sources) and the dielectrics are solved sequentially in each iteration to alleviate the nonlinearity of the problem. In addition, to address the non-uniqueness, regularization methods with different prior assumptions are used to choose a preferred solution. Among them, Tikhonov regularization [8] is used when the objects are assumed to be smooth with few high spatial frequency components. When the objects are assumed to be sparse or group sparse, L_1[9, 10], L_1,2 [11] norm constrained regularizations are used, which help to preserve the edge and sharpness of the object. When the number of distinct dielectric objects is assumed known, this prior knowledge can be introduced with the level-set based optimization method [12] [13]. For the learning based methods, artificial neural networks have been proposed to assist or replace the physical models during the inversion. Those methods have seen improved results at the cost of limited applicability [14-16] or the extra computational cost for training [6][7].

The information used in the microwave imaging methods mentioned above are only limited to the scattered fields measured with a microwave imaging system and the simple prior assumptions enforced with different regularization methods. To utilize more information, another approach is to incorporate the abundant information obtained from other imaging modalities to assist and improve the dielectric image reconstruction, such as [17-20]. Specifically, the Magnetic Resonance (MR) image derived spatial distribution of different types of tissues are incorporated as a spatial prior into the microwave imaging process [19, 20]. The spatial priors are incorporated as soft priors, where pixels/voxels of the same tissue-type have the same weight in the L_2 norm regularized weighting matrix. This method has been tested on numerical 3D breast phantoms [21] and in experimental setups [22]. These methods have shown improved imaging results compared with traditional methods; however, the image tissue type segmentation has to be performed for each new case to get the spatial prior.

SUMMARY

In at least one aspect, a fundamentally different Convolutional Neural Network (CNN) based multi-modality dielectric imaging method is provided. This method can effectively and efficiently incorporate both the structure and the tissue property information acquired from other imaging modalities into the dielectric image reconstruction process. The method includes at least three steps: First, the complex mapping function from MR/Computed Tomography (CT)/Ultrasound images to dielectric images is learned by training a CNN. Then, for each new patient, the trained CNN can take in their MR/CT/Ultrasound images and produce predicted dielectric images. Lastly, the predicted dielectric images are used as the starting or prior images for physics-model-based iterative dielectric image reconstruction. As a result, the CNN-predicted dielectric images significantly alleviate the nonlinearity and ill-posedness of the physics-based inversion model, and the physics-based inversion model will complement the CNN by recovering information missing in the training data and not learned by the CNN. Compared with the previous multi-modality microwave imaging methods, our CNN-based approach has two main advantages: (1) for each new case (new patient), the CNN-based approach can avoid performing complicated tissue segmentation of other modalities' image for information extraction. The relationships between any modality's image data and the corresponding dielectric image is learned and stored in the CNN (2) Besides the information about the spatial distribution of each tissue, the CNN captures more detailed pixel-level tissue inhomogeneity information within each tissue type. Thus, the CNN can extract and transfer more information from other imaging modalities and better assist the physics-model-based microwave imaging. In the previous work, CNN has been used for the estimation of CT, Positron Emission Tomography (PET) images from MR images[23-25]. However, to the best of our knowledge, it is the first time that CNN is utilized to extract prior information from another imaging modality to assist the inverse scattering based dielectric image reconstruction.

In another aspect, a CNN-assisted multi-modality microwave imaging method is provided. The method is demonstrated to be applicable to brain dielectric imaging using MR to train the CNN.

In another aspect, a method for constructing dialect images in a digital medium environment is provided. The method includes a step of training a convolutional neural network to learn a complex mapping function from patient non-dielectric images to dielectric images at a single frequency or multiple frequencies. Characteristically, the convolutional neural network is trained with a test set derived from characterized images. The trained convolutional neural network is applied to a subject's non-dielectric imaging data to determine an initial predicted dielectric image. Finally, the initial predicted dielectric image is applied as a starting input for a physics-model-based iterative dielectric image reconstruction model to form a final predicted dielectric image where non-dielectric imaging data and initial predicted dielectric image can be at the same spatial resolution or at a different spatial resolution.

In still another aspect, a system for constructing dialect images implementing the methods set forth herein. The system includes a computing device configured to receive a subject's non-dielectric imaging data. The computing device includes a computer processor configured execute a trained convolutional neural network by training an untrained convolutional neural network to learn a complex mapping function from patient non-dielectric images to dielectric images at a single frequency or multiple frequencies to form the trained neural network. The untrained convolutional neural network is trained with a test set derived from characterized images. Characteristically, the computer processor is also configured to apply the trained convolutional neural network to the subject's non-dielectric imaging data to determine an initial predicted dielectric image. Finally, the computer processor is also configured to apply the initial predicted dielectric image as a starting input for a physics-model-based iterative dielectric image reconstruction model to form a final predicted dielectric image where non-dielectric imaging data and initial predicted dielectric image can be at the same spatial resolution or at a different spatial resolution.

