BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention pertains to digital pathology, and in particular, to a common framework for finding regions of interest in hematoxylin and eosin (H&E) stained tissue images and for quantifying intratumor cellular spatial heterogeneity in multiplexed/hyperplexed fluorescence tissue images.

2. Description of the Related Art

Digital pathology refers to the acquisition, storage and display of histologically stained tissue samples and is initially gaining traction in niche applications such as: second-opinion telepathology, immunostain interpretation, and intraoperative telepathology. Typically, a large volume of patient data, consisting of 3-50 slides, is generated from biopsy samples and is visually evaluated by a pathologist, under a microscope, but with digital technology by viewing on a high-definition monitor. Because of the manual labor involved, the current workflow practices are time consuming, error-prone and subjective.

Cancer is a heterogeneous disease. In hematoxylin and eosin (H&E) stained tissue images, heterogeneity is characterized by the presence of various histological structures, such as carcinoma in situ, invasive carcinoma, adipose tissue, blood vessels, and normal ducts. One precision medicine approach to both intra- and inter-tumor heterogeneity is to sequence biopsied tissues and to identify a panel of disease-related genomic signatures, specifically for each patient and the distinct regions of a single patient's tumor. However, genomic/epigenomic profiling requires either crushing up tissues or taking multiple core samples from the tumor, yielding a time-consuming and low resolution understanding of the heterogeneity. Spatial interactions between the various histological structures can prognosticate the disease. For example, a tumor nest growing into (invading) blood vessels indicates an increased risk of metastasis. Accurate segmentation of histological structures can thus help build a spatial interaction map to facilitate precision medicine studies combining protein, DNA and RNA biomarkers for deep molecular profiling. This spatial interaction map can also serve as an exploratory tool for pathologists or as a guide to micro-dissection for further molecular profiling.

Histological structure segmentation is a very challenging task because structures such as normal ducts and carcinoma in situ have well-defined boundaries, but many others, such as invasive carcinoma and stroma, do not. Structural morphologies also vary significantly depending on tissue origins (e.g., breast vs. lung), and tissue preparation and staining practices. Historically, biomedical image analysis literature has focused on segmenting nuclei, since nuclei are building blocks for all higher level tissue structures. More recent methods have expanded to segmenting other histological structures, such as the glands in prostate and breast tissue images, with approaches based on nuclei-lumen association, region growth, region-based active contour in combination with Markov Random Field, and deep learning. Some other approaches involve engineering disease- and organ-specific extractors to facilitate analysis of publicly available datasets, such as MITOS (mitotic figures) and GlaS (glands). For example, a typical gland segmentation strategy may involve first identifying a lumen and then searching for the surrounding epithelial layer of cells. However, this strategy is unlikely to work in the case of breast carcinoma in situ, where the duct lumens may be completely filled by tumor cells. A basic mathematical foundation has been developed for supervised segmentation of H&E images, but that foundation has not been tested on more than a couple of examples.

Moreover, for many malignancies, molecular and cellular heterogeneity is a prominent feature among tumors from different patients, between different sites of neoplasia in a single patient, and within a single tumor. Intratumor heterogeneity involves phenotypically distinct cancer cell clonal subpopulations and other cell types that comprise the tumor microenvironment (TME). These cancer cell clonal subpopulations and other cell types include local and bone marrow derived stromal stem and progenitor cells, subclasses of immune inflammatory cells that are either tumor promoting or tumor-killing, cancer associated fibroblasts, endothelial cells and pericytes. The TME can be viewed as an evolving ecosystem where cancer cells engage in heterotypic interactions with these other cell types and use available resources to proliferate and survive. Consistent with this perspective, the spatial relationships among the cell types within the TME (i.e., spatial heterogeneity) appear to be one of the main drivers of disease progression and therapy resistance. Thus, it is imperative to define the spatial heterogeneity within the TME to properly diagnose the specific disease sub-type and identify the optimal course of therapy for individual patients.

To date, intratumor heterogeneity has been explored using three major approaches. The first approach is to take core samples from specific regions of tumors to measure population averages. Heterogeneity in the samples is measured by analyzing multiple cores within the tumor using a number of techniques, including whole exome sequencing, epigenetics, proteomics, and metabolomics. The second approach involves “single cell analyses” using the above methods, RNASeq, imaging or flow cytometry after separation of the cells from the tissue. The third approach uses the spatial resolution of light microscope imaging to maintain spatial context, and is coupled with molecular-specific labels to measure biomarkers in the cells in situ.

Spatial analysis using light microscope imaging facilitates analysis of large areas of tissue sections and/or multiple tumor microarray sections at the cellular and subcellular levels. Subcellular resolution, for example, permits the identification of the activation state of specific biomarkers (e.g. translocation of transcription factors into the nucleus). In addition, recent developments in mass spectrometry imaging permit many cellular constituents to be measured across a tissue section but at a lower resolution than optical microscopy.

Several light microscopy imaging platforms have been developed to characterize cellular biomarker expression levels within tumors including transmitted light and fluorescence. Multivariate information based on fluorescence has been acquired from images of large area tissue sections and tissue microarrays (TMAs) based on DNA, RNA and protein biomarkers, usually from 1 up to 7 fluorescently labeled biomarkers in the same sample (known as multiplexed fluorescence). Multiple commercial platforms can now be used to acquire, process, segment and perform some basic analysis of biomarker signal levels in tissue samples. Recently, platforms have been demonstrated that permit up to 60 fluorescently labeled antibodies and a few DNA or RNA hybridization probes to be acquired in an iterative cycle of labeling, imaging, and quenching fluorescence. It is also now possible to “map” the location of specific cell types, states of cell activations, cell biomarker expression levels and localizations, as well as extracellular constituents in tissue sections and TMAs.

A major challenge is to develop algorithms that can quantify key spatial relationships (interactions, and lack thereof) within the TME, based on panels of biomarkers. Initial efforts in measuring heterogeneity in tissue sections applied diversity metrics from ecological studies, such as Shannon entropy and Rao's quadratic entropy. However, these methods have not been adapted for multiplexed (up to 7 biomarkers) or hyperplexed (>7 biomarkers) immunofluorescence (IF) data. Other methods that account for high dimensional data may not have sophisticated cell phenotyping methods, allowing each biomarker to be only “on” or “off”. Furthermore, few of these methods incorporate the spatial relationships between biomarker patterns in their heterogeneity scores. Indeed, the spatial organization of the TME has been hypothesized to be an important diagnostic biomarker in addition to the expression levels of selected biomarkers from cancer and non-cancer cells.

Other heterogeneity characterization methods have: (i) incorporated spatial information through region of interest sampling without using network-based approaches or taking advantage of multiplexed, (ii) analyzed linear relationships between biomarkers in multiplexed/hyperplexed IF data without considering nonlinear associations or spatial information, and (iii) have characterized multiplexed cell phenotype associations without any characterization of the underlying spatial organization within the tumor. In addition, most other methods report intra-tumor heterogeneity as a single score, thus potentially mapping two spatially different organizations of the TMEs incorrectly to the same score.

There is thus room for improvement in the fields of segmenting H&E images and quantifying intratumor cellular spatial heterogeneity in multiplexed/hyperplexed fluorescence tissue images.

SUMMARY OF THE INVENTION

In one embodiment, a method of identifying regions of interest in an H&E stained tissue image is provided. The method includes receiving image data representing the H&E stained tissue image, quantifying local spatial statistics for the H&E stained tissue image based on the received image data, identifying histological structures within the H&E stained tissue image based on the local spatial statistics, and generating a segmented H&E stained tissue image using the received image data and the identified histological structures.

In another embodiment, a non-transitory computer readable medium is provided that stores one or more programs, including instructions, which when executed by a computer, causes the computer to perform the method just described.

In yet another embodiment, a computerized system for identifying regions of interest in an H&E stained tissue image is provided. The system includes a processing apparatus, wherein the processing apparatus includes: (i) a quantifying component configured for quantifying local spatial statistics for the H&E stained tissue image based on received image data representing the H&E stained tissue image, (ii) an identifying component configured for identifying histological structures within the H&E stained tissue image based on the local spatial statistics, and (iii) a segmented tissue image generating component configured for generating a segmented H&E stained tissue image using the received image data and the identified histological structures.

In still another embodiment, a method of quantifying intratumor cellular spatial heterogeneity in fluorescence tissue images is provided. The method includes receiving image data representing a number of fluorescence tissue images, performing cellular segmentation on the received image data to identify a plurality of cells of the number of fluorescence tissue images, assigning each of the cells to one of a plurality of predetermined biomarker intensity patterns, quantifying spatial statistics for the number of fluorescence tissue images based on the assigned predetermined biomarker intensity patterns, and generating a visual representation of the quantified spatial statistics.

In another embodiment, a non-transitory computer readable medium is provided that stores one or more programs, including instructions, which when executed by a computer, causes the computer to perform the method just described.

In yet another embodiment, a computerized system for quantifying intratumor cellular spatial heterogeneity in fluorescence tissue images is provided. The system includes a processing apparatus, wherein the processing apparatus includes: (i) a cellular segmentation component configured for performing cellular segmentation on image data representing a number of fluorescence tissue images to identify a plurality of cells of the number of fluorescence tissue images, (ii) an assigning component configured for assigning each of the cells to one of a plurality of predetermined biomarker intensity patterns, (iii) a quantifying component for quantifying spatial statistics for the number of fluorescence tissue images based on the assigned predetermined biomarker intensity patterns, and (iv) a visual representation generating component for generating a visual representation of the quantified spatial statistics.

