FIELD

The described embodiments relate to an image diagnostic system, and methods of operating thereof. In some example embodiments, the image diagnostic system, and methods of operating thereof, can relate to histological and histopathological images.

BACKGROUND

Pathologists typically examine tissue specimens prepared on a slide using a microscope. Often, before diagnosing the tissue specimens, pathologists screen the tissue specimens to separate normal tissue from diseased tissue. Diagnosis and classification of the diseased tissue follows thereafter. The screening process can consume a significant amount of time, which delays, and can reduce the amount of time available for, the diagnosis and classification of the specimens. Human error can also impede the screening process.

SUMMARY

The various embodiments described herein generally relate to an image diagnostic system and methods of operating thereof. The disclosed methods and systems can relate to histological images and/or histopathological images.

In accordance with an example embodiment, there is provided a system for identifying one or more tissue types within an image patch according to a hierarchical histological taxonomy. The system includes a supervised digital pathology database having a set of training image patches stored thereon; and a processor in communication with the supervised digital pathology database and operable to: access the set of training image patches stored in the supervised digital pathology database; develop a feature extractor to generate a training feature vector identifying one or more training tissue segments in each training image patch in the set of training image patches; develop a feature classifier to assign a tissue type to each training tissue segment identified in the training feature vector and to update a class confidence score for that tissue type, the tissue type being assigned according to the hierarchical histological taxonomy; apply the feature extractor to the image patch to identify one or more tissue segments within the image patch; and apply the feature classifier to assign a tissue type to the one or more tissue segments identified by the feature extractor for the image patch and to generate a confidence score for assigned tissue type within the image patch with reference to the class confidence score.

In some embodiments, the processor is operable to: develop a convolutional neural network based on the set of training image patches.

In some embodiments, the processor is operable to define an architecture of the convolutional neural network by operating a Network Architecture Search (NAS).

In some embodiments, the feature extractor is generated by applying one of local binary patterns (LBP), pre-trained convolutional neural network, local binary patterns (LBP), and Gabor filters, wavelet filters, and image differential filters.

In some embodiments, the processor is operable to: generate a feature vector identifying the one or more tissue segments of the image patch.

In some embodiments, the processor is operable to: generate a confidence score vector containing the confidence scores for each tissue type identified for the feature vector by the feature classifier.

In some embodiments, the processor is operable to: map one or more prediction models to each tissue segment within the image patch to identify the tissue type relative to the training feature vector and class confidence score for that tissue type.

In some embodiments, the processor is operable to: evaluate the confidence score generated by the feature classifier by comparing the confidence score with a confidence threshold; and generate a quality indicator indicating the confidence score satisfies the confidence threshold, and otherwise, generate the quality indicator indicating the confidence score fails to satisfy the confidence threshold.

In some embodiments, the processor is operable to construct the supervised digital pathology database by: receiving one or more image patches of normal tissue; receiving, via a labelling user interface, one or more user inputs to label each image patch with at least one tissue type according to the hierarchical histological taxonomy; storing each labelled image patch in the supervised digital pathology database.

In some embodiments, the processor is further operable to: select a subset of the training image patches stored in the supervised digital pathology database for verification; and receive, via a verification user interface, one or more user inputs to verify one or more labels associated with each training image patch of the subset the training image patches.

In some embodiments, the processor is operable to: retrieve the set of training image patches from the supervised digital pathology database.

In some embodiments, the one or more image patches are generated from at least one whole slide image of a tissue specimen.

In some embodiments, the processor is further operable to: identify one or more regions of interest in the image patch from which to generate one or more image patches; and identify the one or more tissue types in each image patch of the one or more image patches.

In some embodiments, the system includes an image patch database for storing the one or more image patches.

In accordance with an embodiment, there is provided a method for identifying one or more tissue types within an image patch according to a hierarchical histological taxonomy. The method includes operating a processor to: access a set of training image patches stored in a supervised digital pathology database; develop a feature extractor to generate a training feature vector identifying one or more training tissue segments in each training image patch in the set of training image patches; develop a feature classifier to assign a tissue type to each training tissue segment identified in the training feature vector and to update a class confidence score for that tissue type, the tissue type being assigned according to the hierarchical histological taxonomy; apply the feature extractor to the image patch to identify one or more tissue segments within the image patch; and apply the feature classifier to assign a tissue type to the one or more tissue segments identified by the feature extractor for the image patch and to generate a confidence score for assigned tissue type within the image patch with reference to the class confidence score.

In some embodiments, the method includes operating the processor to: develop a convolutional neural network based on the set of training image patches.

In some embodiments, an architecture of the convolutional neural network includes three convolutional blocks, a global max pooling layer, a single fully-connected layer, and a sigmoid layer.

In some embodiments, the method includes operating the processor to define an architecture of the convolutional neural network by operating a Network Architecture Search (NAS).

In some embodiments, the method includes operating the processor to generate the feature extractor by applying one of pre-trained convolutional neural network, local binary patterns (LBP), Gabor filters, wavelet filters, and image differential filters.

In some embodiments, the method includes operating the processor to: generate a feature vector identifying the one or more tissue segments of the image patch.

In some embodiments, the method includes operating the processor to: generate a confidence score vector containing the confidence scores for each tissue type identified for the feature vector by the feature classifier.

In some embodiments, the confidence score vector has a size corresponding to a number of tissue types identified by the feature classifier for the image patch. The confidence score can include a continuous value.

In some embodiments, the method includes operating the processor to: map one or more prediction models to each tissue segment within the image patch to identify the tissue type relative to the training feature vector and class confidence score for that tissue type.

In some embodiments, the one or more prediction models includes a model based on at least one of a fully-connected (FC) network, a support vector machine (SVM), and a statistical regression.

In some embodiments, the feature classifier is based on a random-forest technique.

In some embodiments, the method includes operating the processor to: evaluate the confidence score generated by the feature classifier by comparing the confidence score with a confidence threshold; and generate a quality indicator indicating the confidence score satisfies the confidence threshold, and otherwise, generate the quality indicator indicating the confidence score fails to satisfy the confidence threshold.

In some embodiments, the quality indicator includes a Boolean value.

In some embodiments, the method includes operating the processor to construct the supervised digital pathology database by: receiving one or more image patches of normal tissue; receiving, via a labelling user interface, one or more user inputs to label each image patch with at least one tissue type according to the hierarchical histological taxonomy; storing each labelled image patch in the supervised digital pathology database.

In some embodiments, the method includes operating the processor to: select a subset of the training image patches stored in the supervised digital pathology database for verification; and receive, via a verification user interface, one or more user inputs to verify one or more labels associated with each training image patch of the subset the training image patches.

In some embodiments, the method includes operating the processor to: retrieve the set of training image patches from the supervised digital pathology database.

In some embodiments, the set of training image patches includes one or more image patches showing at least one normal tissue and the at least one normal tissue is labelled with a tissue type.

In some embodiments, the at least one normal tissue is labelled according to the hierarchical histological taxonomy.

In some embodiments, the method includes operating the processor to generate the one or more image patches from at least one whole slide image of a tissue specimen.

In some embodiments, the method includes operating the processor to: identify one or more regions of interest in the image patch from which to generate one or more image patches; and identify the one or more tissue types in each image patch of the one or more image patches.

In some embodiments, the one or more image patches includes at least one image patch having an overlapping portion with a neighbouring image patch.

In some embodiments, the method includes operating the processor to store the one or more image patches in an image patch database.

In some embodiments, the hierarchical histological taxonomy includes at least two hierarchical levels.

In some embodiments, the hierarchical histological taxonomy includes a set of morphological tissue type labels and a set of functional tissue type labels.

In accordance with an embodiment, there is provided a system for identifying an image patch associated with normal tissue. The system includes a processor operable to: apply a developed feature extractor to the image patch to identify one or more tissue segments within the image patch; apply a developed feature classifier to the one or more tissue segments to identify a respective tissue type and generate a corresponding confidence score representing a degree of similarity according to at least one prediction model generated based on a set of training image patches of a normal tissue of that tissue type, the tissue type being identified according to a hierarchical histology taxonomy; compare each confidence score generated by the feature classifier with a class confidence score generated for the normal tissue to determine whether the image patch is associated with normal tissue; and indicate a health state of each identified tissue type shown in the image patch based on the comparison of the confidence score with the class confidence score.

In some embodiments, the class confidence score includes a confidence score range and the processor is operable determine whether the confidence score falls within the confidence score range.

In some embodiments, the processor is operable to indicate the health state of the identified tissue type is normal when the confidence score satisfies the class confidence score associated to that tissue type, and abnormal when the confidence score fails to satisfy the class confidence score.

In some embodiments, the processor is operable to, in response to determining the health state of the identified tissue type is abnormal, mark the associated image patch to require further processing.

In some embodiments, the processor is operable to, in response to determining the health state of the identified tissue type is abnormal, generate a notification indicating that the associated image patch requires further processing.

In some embodiments, the system further includes a data storage operable to store the image patch identified by the processor to be associated with a normal health state. The data storage can include a normal image patch database for storing the image patch identified by the processor to be associated with the normal health state. The data storage can include an abnormal image patch database for storing the image patch identified by the processor to be associated with an abnormal health state.

In some embodiments, the system further includes a supervised digital pathology database having a set of training image patches stored thereon; and the processor is operable to: receive one or more image patches of normal tissue; receive, via a labelling user interface, one or more user inputs to label each image patch with at least one tissue type according to a hierarchical histological taxonomy; and store each labelled image patch in the supervised digital pathology database.

In some embodiments, the processor is in communication with the supervised digital pathology database and operable to: access the set of training image patches stored in the supervised digital pathology database; develop a feature extractor to generate a training feature vector identifying one or more training tissue segments in each training image patch in the set of training image patches; and develop a feature classifier to assign a tissue type to each training tissue segment identified in the training feature vector and to update a class confidence score for that tissue type, the tissue type being assigned according to the hierarchical histological taxonomy.

In some embodiments, the processor is in communication with the supervised digital pathology database and operable to: develop a convolutional neural network based on the set of training image patches.

In accordance with an embodiment, there is provided a method for identifying an image patch associated with normal tissue. The method includes operating a processor to: apply a developed feature extractor to the image patch to identify one or more tissue segments within the image patch; apply a developed feature classifier to the one or more tissue segments to identify a respective tissue type and generate a corresponding confidence score representing a degree of similarity according to at least one prediction model generated based on a set of training image patches of a normal tissue of that tissue type, the tissue type being identified according to a hierarchical histology taxonomy; compare each confidence score generated by the feature classifier with a class confidence score generated for the normal tissue to determine whether the image patch is associated with normal tissue; and indicate a health state of each identified tissue type shown in the image patch based on the comparison of the confidence score with the class confidence score.

In some embodiments, the class confidence score includes a confidence score range and the method includes operating the processor to determine whether the confidence score falls within the confidence score range.

In some embodiments, the method includes operating the processor to indicate the health state of the identified tissue type is normal when the confidence score satisfies the class confidence score associated to that tissue type, and abnormal when the confidence score fails to satisfy the class confidence score.

In some embodiments, the method includes operating the processor to, in response to determining the health state of the identified tissue type is abnormal, mark the associated image patch to require further processing.

In some embodiments, the method includes operating the processor to, in response to determining the health state of the identified tissue type is abnormal, generate a notification indicating that the associated image patch requires further processing.

In some embodiments, the method includes operating the processor to store the image patch identified to be associated with a normal health state in a data storage.

In some embodiments, the method includes operating the processor to: receive one or more image patches of normal tissue; receive, via a labelling user interface, one or more user inputs to label each image patch with at least one tissue type according to a hierarchical histological taxonomy; and store each labelled image patch in a supervised digital pathology database having a set of training image patches stored thereon.

In some embodiments, the method includes operating the processor to: access the set of training image patches stored in the supervised digital pathology database; develop a feature extractor to generate a training feature vector identifying one or more training tissue segments in each training image patch in the set of training image patches; and develop a feature classifier to assign a tissue type to each training tissue segment identified in the training feature vector and to update a class confidence score for that tissue type, the tissue type being assigned according to the hierarchical histological taxonomy.

In some embodiments, the method includes operating the processor to: develop a convolutional neural network based on the set of training image patches.

In accordance with an embodiment, there is provided a system for generating a pixel-level segmented image patch for an image patch. The system includes a processor operable to: apply a developed feature extractor to the image patch to identify one or more tissue segments within the image patch; apply a developed feature classifier to the one or more tissue segments to identify a respective tissue type and generate a corresponding confidence score representing a degree of similarity according to at least one prediction model generated based on a set of training image patches of a normal tissue of that tissue type, the tissue type being identified according to a hierarchical histology taxonomy; generate a class activation map for each identified tissue type based on the associated confidence score; and apply pixel-level segmentation to the class activation map to define a contour of each tissue segment of that tissue type to generate the pixel-level segmented image patch.

In some embodiments, the processor is operable to apply a gradient-class activation map (Grad-CAM) method to generate the class activation map.

In some embodiments, the processor is operable to: determine one or more tissue characteristics of the tissue type; and generate the class activation map based on the one or more tissue characteristics.

In some embodiments, the one or more tissue characteristics includes one of a morphological tissue type and a functional tissue type.

In some embodiments, the class activation map includes a pixel-level class activation map.

In some embodiments, the processor is operable to apply one or more post-processing techniques to the class activation map to increase a visual homogeneity of each tissue type segment.

In some embodiments, the one or more post-processing techniques includes a fully-connected conditional random field (CRF) modeling method.

In some embodiments, the processor is operable to adjust the class activation map with a background class activation map generated based on image data associated with high white-illumination values.

In some embodiments, the processor is operable to adjust the class activation map associated with a functional tissue type with a non-tissue class activation map generated based on image data associated with non-tissue characteristics.

In some embodiments, the system includes a supervised digital pathology database having a set of training image patches stored thereon; and the processor is operable to: receive one or more image patches of normal tissue; receive, via a labelling user interface, one or more user inputs to label each image patch with at least one tissue type according to a hierarchical histological taxonomy; and store each labelled image patch in the supervised digital pathology database.

In some embodiments, the processor is in communication with the supervised digital pathology database and operable to: access the set of training image patches stored in the supervised digital pathology database; develop a feature extractor to generate a training feature vector identifying one or more training tissue segments in each training image patch in the set of training image patches; and develop a feature classifier to assign a tissue type to each training tissue segment identified in the training feature vector and to update a class confidence score for that tissue type, the tissue type being assigned according to the hierarchical histological taxonomy.

In some embodiments, the processor is in communication with the supervised digital pathology database and operable to: develop a convolutional neural network based on the set of training image patches.

In accordance with an embodiment, there is provided a method for generating a pixel-level segmented image patch for an image patch. The method includes operating a processor to: apply a developed feature extractor to the image patch to identify one or more tissue segments within the image patch; apply a developed feature classifier to the one or more tissue segments to identify a respective tissue type and generate a corresponding confidence score representing a degree of similarity according to at least one prediction model generated based on a set of training image patches of a normal tissue of that tissue type, the tissue type being identified according to a hierarchical histology taxonomy; generate a class activation map for each identified tissue type based on the associated confidence score; and apply pixel-level segmentation to the class activation map to define a contour of each tissue segment of that tissue type to generate the pixel-level segmented image patch.

In some embodiments, the method includes operating the processor to apply a gradient-class activation map (Grad-CAM) method to generate the class activation map.

In some embodiments, the method includes operating the processor to: determine one or more tissue characteristics of the tissue type; and generate the class activation map based on the one or more tissue characteristics.

In some embodiments, the method includes operating the processor to apply one or more post-processing techniques to the class activation map to increase a visual homogeneity of each tissue type segment.

In some embodiments, the method includes operating the processor to adjust the class activation map with a background class activation map generated based on image data associated with high white-illumination values.

In some embodiments, the method includes operating the processor to adjust the class activation map associated with a functional tissue type with a non-tissue class activation map generated based on image data associated with non-tissue characteristics.

In some embodiments, the method includes operating the processor to: receive one or more image patches of normal tissue; receive, via a labelling user interface, one or more user inputs to label each image patch with at least one tissue type according to a hierarchical histological taxonomy; and store each labelled image patch in a supervised digital pathology database having a set of training image patches stored thereon.

In some embodiments, the method includes operating the processor to: access the set of training image patches stored in the supervised digital pathology database; develop a feature extractor to generate a training feature vector identifying one or more training tissue segments in each training image patch in the set of training image patches; and develop a feature classifier to assign a tissue type to each training tissue segment identified in the training feature vector and to update a class confidence score for that tissue type, the tissue type being assigned according to the hierarchical histological taxonomy.

In some embodiments, the method includes operating the processor to: develop a convolutional neural network based on the set of training image patches.

In accordance with an embodiment, there is provided a system for generating an encoded image patch for an image patch of at least one tissue. The system includes any system disclosed herein for generating a pixel-level segmented image patch of the image patch; and a processor operable to: generate one or more binary masks for the image patch, each binary mask being associated with an identified tissue type and corresponding to a pixel-level representation of one or more tissue segments of that tissue type; for each binary mask, identify one or more features related to a structure of each tissue segment; generate one or more feature correspondence values relating each feature identified in each binary mask and each tissue type identified for the image patch; and generate the encoded image patch based on the one or more feature correspondence values.

In some embodiments, the processor is operable to generate the feature correspondence value to represent a similarity of a structure of a tissue segment to a feature of the one or more features.

In some embodiments, the encoded image patch includes a matrix of the one or more feature correspondence values with a number of columns corresponding to a number of unique features identified in the one or more binary masks and a number of rows corresponding to a number of unique tissue types identified for the image patch.

In some embodiments, the processor is operable to generate a unique encoded image patch for each hierarchy level of the hierarchical histological taxonomy based on which the one or more tissue types are identified.

In some embodiments, the processor is operable to aggregate each unique encoded image patch to generate an aggregated encoded image patch.

In accordance with an embodiment, there is provided a method for generating an encoded image patch for an image patch of at least one tissue. The method includes operating a processor to: generate a pixel-level segmented image patch of the image patch according to any method disclosed herein; generate one or more binary masks for the image patch, each binary mask being associated with an identified tissue type and corresponding to a pixel-level representation of one or more tissue segments of that tissue type; for each binary mask, identify one or more features related to a structure of each tissue segment; generate one or more feature correspondence values relating each feature identified in each binary mask and each tissue type identified for the image patch; and generate the encoded image patch based on the one or more feature correspondence values.

In some embodiments, the method includes operating the processor to generate the feature correspondence value to represent a similarity of a structure of a tissue segment to a feature of the one or more features.

In some embodiments, the method includes operating the processor to generate a unique encoded image patch for each hierarchy level of the hierarchical histological taxonomy based on which the one or more tissue types are identified.

In some embodiments, the method includes operating the processor to aggregate each unique encoded image patch to generate an aggregated encoded image patch.

In accordance with an embodiment, there is provided a system for searching for one or more histological histopathological images. The system includes: a histopathological histological image databank constructed using any system disclosed herein; and a processor operable to: receive a search query defining one or more histopathological histological characteristics; and conduct a search of the histopathological histological databank for one or more encoded image patches that satisfies the search query.

In accordance with an embodiment, there is provided a method for searching for one or more histopathological images. The method includes operating a processor to: construct a histopathological image databank according to any method disclosed herein; receive a search query defining one or more histopathological characteristics; and conduct a search of the histopathological databank for one or more encoded image patches that satisfies the search query.

In accordance with an embodiment, there is provided a system for assigning an image patch to one or more pathological cases. The system includes: a histopathological histological image databank having a set of encoded image patches stored thereon and each encoded image patch is associated with a pathological case of the one or more pathological cases; and a processor in communication with the histopathological histological image databank and operable to: receive a whole slide image and divide at least a portion of the whole slide image into a plurality of image patches; determine using any system disclosed herein whether any image patch of the plurality of image patches is associated with abnormal tissue; in response to determining that an image patch of the plurality of image patches is associated with abnormal tissue, generate an encoded image patch of the image patch using any system disclosed herein; and compare the encoded image patch with the set of encoded image patches stored in the histopathological histological image databank to assign at least one pathological case to the encoded image patch.

In some embodiments, the processor is operable to construct the histopathological histological image databank by: receiving a set of training whole slide images, each training whole slide image associated with the one or more pathological cases; receiving, via a labelling user interface, user inputs associating at least one region within each training whole slide image with the one or more pathological cases; for each training whole slide image, generating a plurality of training image patches for the at least one region associated with the one or more pathological cases; generating an encoded training image patch using any system disclosed for each training image patch; and storing the encoded training image patch in the histopathological histological image databank in association with the associated one or more pathological cases.

In some embodiments, the processor is operable to store the encoded training image patch in one or more pathological case databases, each pathological case database being associated with a pathological case.

In accordance with an embodiment, there is provided a method for assigning an image patch to one or more pathological cases. The method includes operating a processor to: a histopathological image databank having a set of encoded image patches stored thereon and each encoded image patch is associated with a pathological case of the one or more pathological cases; and receive a whole slide image and divide at least a portion of the whole slide image into a plurality of image patches; determine whether any image patch of the plurality of image patches is associated with abnormal tissue according to any of the methods disclosed herein; in response to determining that an image patch of the plurality of image patches is associated with abnormal tissue, generate an encoded image patch of the image patch according to any method disclosed herein; and compare the encoded image patch with the set of encoded image patches stored in the histopathological image databank to assign at least one pathological case to the encoded image patch.

In some embodiments, the method includes operating the processor to: receive a set of training whole slide images, each training whole slide image associated with the one or more pathological cases; receive, via a labelling user interface, user inputs associating at least one region within each training whole slide image with the one or more pathological cases; for each training whole slide image, generate a plurality of training image patches for the at least one region associated with the one or more pathological cases; generate an encoded training image patch according to any method disclosed herein for each training image patch; and store the encoded training image patch in the histopathological image databank in association with the associated one or more pathological cases.

In some embodiments, the method includes operating the processor to store the encoded training image patch in one or more pathological case databases, each pathological case database being associated with a pathological case.

In some embodiments, the one or more pathological cases includes one or more disease types.

In accordance with some embodiments, there is provided a non-transitory computer-readable medium having instructions executable on a processor for implementing any one of the methods disclosed herein.

