CROSS-REFERENCE TO PRIOR APPLICATIONS

This application is the U.S. National Phase application under 35 U.S.C. § 371 of International Application No. PCT/EP2016/069255, filed on Aug. 12, 2016, which claims the benefit of European Patent Application No. 15182141.0, filed on Aug. 24, 2015. These applications are hereby incorporated by reference herein.

FIELD OF THE INVENTION

The invention relates to a server-client architecture, in particular to the server and the client as such, and a method for handling and processing of digital pathology data.

BACKGROUND OF THE INVENTION

A pathologist has a central role in the diagnostic part of a care cycle. When a biologic material sample is taken from a patient during biopsy, it is usually a pathologist who makes the final diagnosis as to the nature and the gravity of a potential disease by microscopic inspection of the tissue and cells from the biopsy.

In an example, a slide for the microscopic inspection can be prepared in several steps as follows. First, the biological material sample can be placed in formalin for fixation and subsequently processed into a paraffin block, from which thin, about 5 μm, slices are cut. One of these slices is then placed on a substrate of the slide, in particular on a glass substrate, after which one or more stained substances may be applied, such that relevant cell or tissue parts are visible with the microscope. Thereafter, a match fluid and/or a fixative may be added to the slice. Further, a thin, about 170 μm, transparent cover slip can be placed on top of the slice, such that the slice is sealed. This enables a long-term storage, in particular more than 10 years, of the slide including the slice.

Although pathology is typically an analog profession, there has been recently a move towards digital pathology, in order to improve the diagnostic efficiency and quality. Digital pathology refers to a digitalization of the information needed, for analyzing a biological material, in particular a slice, in the field of biology, histology and/or pathology. In particular, digital pathology refers to a digital molecular diagnostic of slices, which have been previously cut from a biological material sample. An exemplary adaptation of the digital pathology would result in the pathologist handling the slices no longer himself physically, but instead to work with the digital images, which can be taken from the slides and other clinical information for making his or her diagnosis.

Document US 2008/0112614 A1 relates to a method for identifying a region of interest within a digital image. The method includes receiving the digital image. The image is divided into a plurality of tiles. It is determined, which of the plurality of tiles includes the region of interest. A graphics processing unit calculates a pixel intensity histogram for the tiles that have been determined to include the region of interest. Not calculated is a pixel intensity histogram for the tiles that do not include the region of interest. Image processing results may then be displayed or printed. A medical practitioner, for example a radiologist, may then examine the processing results.

For the calculation of intensity histograms, a high performance CPU may be needed. Limiting the calculation effort for calculating the histograms to the tiles, that have been determined to include the region of interest, may reduce the calculation effort. However, from a practical point of view, a medical practitioner still needs to wait until the CPU has finished the calculation, before the results can be displayed.

SUMMARY OF THE INVENTION

There may be a need to facilitate the responsiveness of data related to a region of interest of a digital image of a slice of biological material.

The object of the invention is solved by the subject-matter of the independent claims, wherein further embodiments are incorporated in the dependent claims.

According to a first aspect of the invention, a server is provided, comprising a server input interface, a server processing unit, a server storage unit and a server output interface. The server input interface is configured to receive a digital image of a slice of biological material. The slice of a biological material has been applied with at least one staining substance. The server processing unit is configured to classify each pixel of the digital image, wherein a pixel is classified with respect to each staining substance as stained, in case a color-property of the pixel is within a predefined property-range indicative for an absorption of the respective staining substance. The server storage unit is configured to store the digital image and the classification of the pixels of the digital image. The server input interface is configured to receive a request indicating a region of interest of the digital image from the client. The server output interface is configured to provide the stored classification and/or other data related to the classification of the pixels at the region of interest to the client.

As a result, the classification can be carried out and/or finished previously to the request. In case the request occurs, relevant data, in particular the classification stored, may be provided efficiently without the need to compute these at the time of the request or thereafter. Since each pixel of the digital image is classified, the availability of the data related to any region of interest of the digital image is fulfilled. Accordingly, independent of a choice of the region of interest of the digital image, relevant data related to the classification of the pixels, in particular at the region of interest, may be provided in return of the request. Due to this high availability of data related to the classification of the pixels at the region of interest, the server comprises a high responsiveness. A client, which may have requested for the relevant data, may be provided instantly with the requested data, such that the performance at the client site may be increased.

In an example, the server may relate to a computer or a computer system.

In an example, the term “slice” relates to a cut from a biological sample material in form of a thin, in particular about 3 to 10 μm, slice.

In an example, the term “staining substance” relates to a substance enabled to enhance a contrast and/or highlight structures in biological material, in particular of a slice of the biological material.

In an example, the term “staining substance” may also relate to a fluorescent substance, also called as fluorescent staining substance, enabled to fluorescently label structures and/or molecules in biological material, in particular of a slice of the biological material.

In an example, Hematoxylin (H), Eosin (E) or a combination of Hematoxylin and Eosin (H&E) may be employed as a staining substance for staining a slice.

In an example, a biological fluorescent stain, an immunofluorescence and/or a fluorescent protein may be employed as a staining substance for staining a slice.

In an example, staining a slice allows to distinguish for instance between cells, tissue and interconnections, in particular between muscle fibers, connective tissue, blood cells and/or organelles within individual cells.

In an example, the slice of a biological material can be applied with more than one staining substance. For instance, the slice may be applied with a first staining substance and a second staining substance. The first staining substance can be called the staining substance and the second staining substance can be called the counter-staining substance.

In an example, the digital image of the slice comprises a plurality of pixels. For instance, a digital image may comprise more than 100,000 pixels.

In an example, staining a slice may change its color property, which accordingly has influence on the color property of the digital image. It should be understood that the color property may also relate to a fluorescence property. Therefore, a color-property of each pixel of the digital image may be used, to classify, whether the respective pixel shows a small area of the slice, which has been applied with the at least one staining substance. In an example, the classification of a pixel of the digital image relates to a determination whether a color-property of the pixel is of a staining substance specific property-range. The property-range can be predefined, in particular for each of the at least one staining substance. Accordingly, for each staining substance, a specific predefined property-range can be provided.

In an example, the hue of a pixel forms a color-property of the respective pixel.

In an example, the property-range may be formed by a color-band.

In an example, an absorption of a staining substance may relate to a binding of at least a part of a staining substance to structures and/or molecules of a biological material. For instance, a fluorescence staining substance as the staining substance may bind particular molecules of the biological material. Hematoxylin as another example for the staining substance may bind particular salts with acidic of the biological material.

In an example, a slice of biological material at least partly absorbs the applied staining substance. In turn of the absorption of the staining substance, the hue of the slice may change, where the absorption has taken place. Moreover, the degree of the hue-change may depend on the degree of the absorption of the staining substance. Accordingly, a color-band may be indicative for an absorption of a specific staining substance. Thus, different staining substances may relate to different color-bands.

In an example, classifying each pixel of the digital image, wherein a pixel is classified with respect to each staining substance as stained, results in the classification of the pixels of the digital image. The “classification” of the pixels of the digital image may refer to as the classification result.

In an example, the server input interface receives a request indicating a region of interest of the digital image from a client.

In an example, a client relates to a computer, a computer system and/or a software unit. A client may be configured to access a server via a computer network.

In an example, the region of interest (ROI) relates to a region of the digital image, which may show a corresponding region of the slice, which may be for instance according to a user of the client, preferably a pathologist, the relevant part of the slice over which a further analysis or calculation should be carried out. This calculation may comprise a calculation of a score to characterize a staining of the region of interest.

In an example, the request may indicate an arbitrary region of interest of the digital image. Since each pixel has been classified, and the classification result has been stored, the server can be configured to provide a classification result related to the region of interest to the client instantly. The classification result related to the region of interest may be the data related to the classification of the pixels at the region of interest.

In an example, the term “data related to the classification of the pixels at the region of interest” relates to data, which has been calculated—directly or indirectly—on the basis of the results of the classification of the pixels at the region of interest. Preferably, the calculation has been performed prior the request of the client. In this case, the data is pre-calculated and/or pre-processed.

In an example, the term “data related to the classification of the pixels at the region of interest” may also be referred to as “data related to the region of interest”

As a result, pre-processed data regarding the region of interest may be provided for further purposes with high responsiveness from the server to the client. A pathologist at the client site may manually indicate the region of interest on a copy of the digital image. The availability of the classification result according to a certain staining at the server site and the configuration of the server to be able to provide data related to the classification of the pixels at the region of interest, in particular the classification of the pixels at the region of interest, instantly to the client, reduces the processing time at the client for further calculation, in particular for calculating a score. Accordingly, the client can provide a higher performance to the user of the client.

According to an exemplary embodiment of the invention, the server output interface is configured to provide at least a part of the data stored by the server storage unit as the data related to the classification of the pixels at the region of interest to the client.

In an example, the classification stored by the server storage unit is provided as the data related to the classification of the pixels at the region of interest. As a result, the pre-calculated classification or other data stored, each having a direct or indirect relation to the classification of the pixels at the region of interest, may be provided to the client instantly after the request. The client does not necessarily need the processing power for such a calculation, for instance to calculate the classification of each pixel of the digital image. Thus, the performance provided by the client to the user of the client can be increased.

According to an exemplary embodiment of the invention, the server processing unit is configured to calculate at each pixel of the digital image for each staining substance an absorption value on the basis of the color-property of the respective pixel.

According to an exemplary embodiment of the invention, the server processing unit is configured to determine a plurality of digital tiles, each associated with pixels at a sub-area of a total area of the digital image, such that each pixel of a majority of the pixels of the digital image is associated by at least two tiles. The server processing unit is configured to quantize the at least one absorption value of each pixel of the digital image into one of a predefined number of absorption levels. The server processing unit is configured to calculate for each tile an absorption frequency vector for each staining substance, wherein the absorption frequency vectors comprise for each absorption level an entry indicating for the respective staining substance the number of stained pixels of the tile at the respective absorption level. The server storage unit is configured to store the digital tiles and/or the absorption frequency vectors.

In an example, each tile corresponds to a digital sub-image of the digital image.

In an example, adjacent tiles may overlap, in which case they are associated with a common set of pixels of the digital image.

In an example, the absorption values are quantized to form values in correspondence with the absorption levels. The number of absorption levels is limited, such that the processing effort can be reduced for the further steps to be carried out by the server processing unit.

In an example, more than one staining substance has been applied to the slice of biological material. Accordingly, a digital tile associated with a sub-area of the digital image of the slice may comprise pixels classified as stained by a first staining substance and also pixels classified as stained with a second staining substance, etc. Accordingly, for each tile an absorption frequency vector can be calculated for each staining substance.

In an example, each absorption frequency vector relates to one of the staining substances. Each entry of the absorption frequency vector may relate to the number of staining substance specific stained pixels at a corresponding absorption level. For each tile, more than one absorption frequency vector can be provided, in particular as many absorption frequency vectors as staining substances may have been applied to the slice of biological material. The number of entries of the frequency vectors preferably corresponds to the number of absorption levels.

In an example, an entry of a frequency vector relates to the number of pixels classified with the associated staining substance at the tile, which the respective frequency vector relates to.

As a result, an absorption frequency vector provides a distribution frequency of a respective staining substance specific to classified pixels at different absorption levels. Thus, an absorption frequency vector comprises a compact representation about the distribution of the staining substance at a sub-area of the digital image corresponding to the tile, which the absorption frequency vector relates to.

In an example, the server processing unit may be configured to calculate the absorption frequency vectors previously to the request of data related to a specific region of interest. Thus, the calculation of the absorption frequency vectors can be finished, before a client may send a request for data related to a classification of the pixels at the region of interest. Thus, in case the request occurs, relevant data related to the classification of the pixels at the region of interest can be provided initially.

In an example, data related to the classification of the pixels at the region of interest may relate to the absorption frequency vectors, in particular to a sub-set of the absorption frequency vectors. Thus, the absorption frequency vectors, in particular a sub-set thereof, can be provided to the client initially, in return of the request. This initial return increases the performance provided to a user at the client site.

According to an exemplary embodiment of the invention, the server processing unit is configured to determine the plurality of tiles in a quadtree structure and/or an integral image structure.

In an example, tiles at different quadtree structure-levels may have different size and/or are associated with different sizes of sub-areas of the digital image.

In an example, tiles of different quadtree structure-levels may overlap. Accordingly, redundant information may be provided by the tiles at the different quadtree structure-levels.

According to an exemplary embodiment of the invention, the server processing unit is configured to determine upon the request a minimum number of tiles of the plurality of stored digital tiles as selected tiles, such that the selected tiles forming form an integral area corresponding to the region of interest of the digital image. The server input interface is configured to receive for each staining substance an absorption threshold value for the region of interest. The server processing unit is configured to select upon the request the entries of the stored absorption frequency vectors of the selected tiles as selected entries, which related absorption levels are each larger than their related absorption threshold value. The server processing unit is configured to calculate upon the request a score, characterizing a staining at the region of interest, as a function of the selected entries and/or all entries of the stored absorption frequency vectors of the selected tiles. The server output interface is configured to provide the score as the data related to the classification of the pixels at the region of interest to the client.

In an example, the selected tiles may have different sizes.

In an example, adjacent selected tiles may overlap or border perfectly at each other.

In an example, tiles associated with a central region of the region of interest of the digital image may have a larger size than selected tiles associated with another region at the outer edge of the region of interest of the digital image.

As a result, the central region of the region of interest may be covered with less, but larger tiles.

In an example, the average size of the selected tiles may be as large as possible, in order to provide a minimum number of tiles forming the integral area of the region of interest.

In an example, the absorption threshold value may relate to a value for the absorption level or absorption value of a pixel above which the pixel may be considered for calculating the score, characterizing a staining at the region of interest.

As a result, the absorption threshold value may be chosen suitably to neglect pixels of the region of interest having less influence on the actual staining character and/or a respective score.

In an example, the selected entries of a frequency vector relate to the absorption levels of the pixels of the region of interest being larger than or equal to the related absorption threshold value. The relation may refer to the staining substance, to which the absorption frequency vector is associated with.

In an example, for each staining substance, a score can be calculated for the region of interest.

In an example, for a combination of all staining substances, a score can be calculated for the region of interest.

In an example, a score characterizes a staining of a region of interest with a staining substance. Accordingly, for the region of interest, more than one score, in particular as many scores as number of staining substances applied to the slice, may be provided.

In an example, the server output interface is configured to provide the at least one score as the data related to the classification of the pixels at the region of interest to the client.

As a result, the server processing unit may be employed to identify the relevant entries of the absorption frequency vectors, wherein the processing unit may be further configured to compute a score as a function of the identified entries of the absorption frequency vectors, such that the score may be provided to the client.

In an example, the server processing unit may be configured to calculate the absorption frequency vectors prior to the request for data related to a classification of the pixels at the region of interest.

In a further example, the server processing unit may be configured to calculate upon the request, and thus preferably after the request has been received by the server, the at least one score.

As a result, the server processing unit may be configured to calculate the last processing step in turn of the request according to an individual region of interest with a related absorption level provided to the server.

As a result, the server may have pre-processed most of the steps previously to the receipt of the request. As the server may have large processing power, the at least one score may be calculated and thereafter provided to the client within a very limited time. As a result, the server has a high responsiveness.

According to a second aspect of the invention, a client is provided, that comprises a client input interface, a user interface and a client output interface. The client input interface is configured to receive a digital image of a slice of biological material, which has been applied with at least one staining substance. The user interface is configured to display the digital image. The user interface is configured to receive a selection signal from a user and to select a region of interest of the digital image on the basis of the selection signal. The client output interface is configured to send a request for data related to a classification of the pixels at the region of interest to a server. The client input interface is configured to receive upon the request the data related to the classification of the pixels at the region of interest from the server.

In an example, the client relates to a computer unit or a software unit. The client may have access to the server via a computer network.

In an example, the digital image received by the client corresponds to the digital image received by the server. Accordingly, the images may relate to the same digital image.

In an example, the user interface may comprise a display. In particular, the user interface may comprise a touch-screen.

In an example, the user interface may comprise a graphical user interface.

In an example, the user interface is configured to receive a signal from a keypad, a mouse and/or any other computer peripheral that allows a user to enter a user input.

As a result, a user can use the client to send a request for data related to a classification of the pixels at the region of interest to the server, which preferably has pre-calculated data related to the classification of the pixels at the region of interest and may be configured to send this data to the client initially in return. Thus, a client input interface may receive the data related to the classification of the pixels at the region of interest with a short delay after sending the request. Accordingly, the client can provide a high performance and/or responsiveness.

According to a third aspect of the invention, a system is provided, comprising a server according to the first aspect of the invention, and at least one client according to the second aspect of the invention.

According to a fourth aspect of the invention, a method is provided, comprising the following steps:

a) Receiving at a server a digital image of a slice of a biological material, which has been applied with at least one staining substance.

b) Classifying at the server each pixel of the digital image, wherein a pixel is classified with respect to each staining substance as stained, in case a color-property of the pixel is within a predefined property-range indicative for an absorption of the respective staining substance.

c) Receiving at a client the same digital image of the slice of a biological material.

d) Displaying the digital image received at the client by a user interface of the client.

e) Selecting a region of interest of the digital image at the client upon receiving and on the basis of a selection signal provided by a user via the user interface.

f) Sending a request for data related to the classification of the pixels at the region of interest from the client to the server.

g) Sending the classification and/or other data related to the classification of the pixels at the region of interest of the digital image from the server to the client, in case the classifying of step b) has been finished before receiving the request for data related to the classification of the pixels at the region of interest from the client.

In an example, a computer program element for controlling a system according to the third aspect of the invention is provided, which, when being executed by the server processing unit and the at least one client unit, is adapted to perform the method according to the fourth aspect of the invention.

In an example, a computer-readable medium having stored the program element is provided.

According to an aspect of the invention, a client-server-system is provided. The client and the server are provided with the same digital image of a slice of a biological material, which has been applied with a staining substance. Usually, the server comprises a large processing power. Accordingly, the server is used for pre-processing the digital image and provides results of the pre-processing to the client in turn of a request. The server comprises a server processing unit, which is configured to classify each pixel of the digital image, wherein the pixel is classified as stained, in case a color-property of the pixel characterizes a spot corresponding to the pixel at the slice as being stained with the staining substance, whereas the slice has absorbed the staining substance at the respective spot. Thus, all pixels of the digital image are classified. A sub-set of all pixels is classified as being stained. The remaining pixels may be classified as undefined, unstained or stained with a different substance. The server processing unit can be configured to classify the remaining pixels respectively.

Further, the server comprises a server storage unit, to store the digital image and the results of the classification of the pixels. The classification results are called the “classification of the pixels”. In case a user at the client site wants to know, whether a region of interest at the digital image comprises pixels, which have been classified as stained, the client is configured to receive a selection signal of the user via an user interface. A client processing unit of the client is configured to determine the region of interest at the digital image. Thus, the client is configured to request data related to the classification of the pixels at the region of interest at the server. The server has pre-processed the classification of the digital image and/or further data based thereon and is thus initially prepared to provide the data related to the classification of the pixels at the region of interest to the client. Accordingly, the server can provide the classification results as the data related to the classification of the pixels at the region of interest at the digital image to the client. The client can display the received classification results, preferably with a display of a user interface. Consequently, the user can see within a short time delay, whether pixels at the region of interest of the digital image are classified as stained.

Further, the server can be configured to determine on the basis of the classification and an absorption threshold value, whether pixels classified as stained at the region of interest contribute to a score, which characterize the staining of the region of interest of the digital image. The client can be configured to receive a signal from the user in order to adapt the absorption threshold value and to send it to the server, whereas at the server the score can be updated accordingly. The score calculated at the server can be taken as the data related to the classification of the pixels at the region of interest and provided to the client. Thus, the client user interface may display the score.

Moreover, the server processing unit may calculate absorption frequency vectors, which each relate to a tile. A plurality of tiles and accordingly a plurality of absorption frequency vectors are provided. Each of the plurality of tiles is associated with a sub-image of the digital image. Furthermore, the tiles are overlapping each other. In particular, the tiles relate to a quadtree structure. Therefore, the tiles are categorized in different levels, whereas tiles of different levels are overlapping each other. The tiles at the same level are adjacent to each other. Accordingly, tiles of different level comprise redundant information about the digital image. Each tile comprises a plurality of pixels.

A sub-set of these pixels may be classified as stained. Depending on the color-property of the stained pixels, a quantization of the stained pixels may be determined, in order to calculate entries for the absorption frequency vectors in correspondence with the number of the stained pixels at the respective absorption levels and a related tile. The absorption levels are limited to the number of entries of the vectors. The absorption frequency vectors are pre-processed at the server processing unit and stored at the server storage unit. In case a request from the client occurs at the server for data related to a region of interest, the server processing unit can select a group of a minimum number of tiles from the different levels. Accordingly, in the center of the region of interest, larger tiles are selected whereas at the borders of the region of interest, smaller tiles are selected. The selected tiles form an integral image corresponding to the region of interest. Since the absorption frequency vectors for the selected tiles have been pre-processed, the relevant information relating to the region of interest can be provided by the respective absorption frequency vectors to the client. The client can select the entries of the absorption frequency vectors of the tiles on the basis of the absorption threshold value. For instance, just the half of the entries of each absorption frequency vector may be taken to calculate at the client a score for the region of interest, whereas the score characterizes the staining of the pixels having an absorption value larger than the absorption threshold value. This gives the user the ability, to update the absorption threshold value at the client site, while the client processing unit in turn may calculate the score on the basis of the predetermined absorption frequency vectors and the (updated) absorption threshold value. Thus, a higher performance can be provided to the client, since the absorption frequency vectors, as the basis for the calculation of the score, have been processed for a plurality of different sets of absorption levels.

These and other aspects of the invention will be apparent from and be elucidated with reference to the embodiments described hereinafter.

