CROSS REFERENCE TO RELATED APPLICATION

The disclosure of Japanese Patent Application No. 2020-012483 filed on Jan. 29, 2020 including specification, drawings and claims is incorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to an image processing technique for processing an original image captured by imaging a pathological specimen. Particularly, this invention relates to a technique for supporting a diagnosis by a pathologist by reflecting and presenting pathological findings extracted by an image analysis on a displayed image.

2. Description of the Related Art

At a pathological diagnosis site, a skilled doctor (pathologist) has observed a pathological specimen collected from a patient and diagnosed by comprehensively judging a state of the pathological specimen. Such an operation is a large load on the pathologist and possibly causes a variation in diagnosis result. Thus, the development of a technique for mechanizing and automating this operation has been in progress and many techniques to partially replace a diagnosis operation of a doctor have been proposed. For example, in techniques described in WO 2013/027399 and WO 2013/024600, a part having a specific morphological feature is extracted from an image captured by imaging a pathological specimen by an image analysis and a result quantitatively or statistically evaluating an analysis result is presented.

Particularly in recent years, researches on artificial intelligence and the application thereof have advanced to a practical level. This has enabled even comprehensive judgment, which had to be conventionally made based on experiences by humans and is based on information not necessarily suitable for quantification, to be automated.

However, at the moment, it has not yet reached to automate an entire operation performed by a pathologist for diagnosis, and judgment based on the pathologist's experiences remains to be necessary. Further, in many countries, it is legalized that a final diagnostic action needs to be performed by a doctor.

Many pieces of information including those not suitable for quantification can be obtained from an image of a pathological specimen. However, to which piece of information importance should be placed and evaluation criteria for the important piece of information are still finally left up to the knowledge and experiences of a doctor. Thus, the presentation of quantitative information on some of indices as in the above conventional art cannot be necessarily said to be high in versatility.

Then, it cannot be necessarily said to be effective to present only information quantified as indices on judgment and an automatic judgment result based on this quantified information and judgment criteria determined in advance. At the moment, it is rather more effective in some cases to present many pieces of information obtained from an image as they are without any prejudgment and in various modes in response to a request of a user (doctor) in a multifaceted way. Thus far, the user has obtained pathological findings by observing a specimen image in detail and diagnosed based on that result. If a series of such operations can be supported from the aspect of extracting and presenting pathological findings, a load on a doctor relating to a diagnosis can be drastically reduced and a stable diagnosis can be enabled.

A pathological specimen is generally stained with HE (Hematoxylin and Eosin). This staining method can separately stain cytoplasms and cell nuclei, but is not for selectively staining specific cell species, disease sites and the like. Due to this universality, various pieces of information can be extracted from an image of a stained specimen. On the other hand, deep knowledge is necessary to effectively extract information on a specific disease. By mechanically replacing this operation, it is expected that the user can be dedicated to a diagnosis based on the extracted piece of information and an efficient and stable diagnosis result can be obtained.

An image processing system for supporting a diagnosis operation in a multifaceted way from these perspectives has not been proposed thus far. Specifically, many of conventional techniques focus on features unique to specific cells and diseases and utilize these features, and a range of its application has been restrictive.

SUMMARY OF THE INVENTION

This invention was developed in view of the above problem and an object thereof is to provide a technique capable of effectively supporting a diagnosis operation by a doctor by presenting various pieces of information obtained from an image easily and in a multifaceted way in displaying the image captured by imaging a pathological specimen.

To achieve the above object, one aspect of an image processing apparatus according to this invention includes an image acquirer for obtaining an original image captured by imaging a pathological specimen, an information acquirer for obtaining finding information relating to pathological finding included in the original image for at least one type of the pathological finding, an image generator for, using at least a part of the original image as a source image, generating an output image for screen display including a processed image obtained by superimposing visual information corresponding to the pathological finding specified by the finding information on the source image as an image element, and a receiver for receiving an operation input from a user. Here, the image generator sets a relative density between the source image and the visual information in the processed image according to the operation input and has a plurality of processing modes including a first mode and a second mode for generating the output image, the processing modes being selectable by the operation input.

Further, to achieve the above object, one aspect of an image processing method according to this invention includes obtaining an original image captured by imaging a pathological specimen, obtaining finding information relating to pathological finding in the original image for at least one type of the pathological finding, receiving an operation input from a user, and generating an output image for screen display by using at least a part of the original image as a source image, the output image including a processed image obtained by superimposing visual information representing positions of the pathological finding specified by the finding information on the source image as an image element. Here, a relative density between the source image and the visual information in the processed image is set according to the operation input and a processing mode for generating the output image is selected from a plurality of processing modes including a first mode and a second mode for generating the output images by the operation input.

In these inventions, the plurality of processing modes include a first mode and a second mode. The first mode is the processing mode for generating the output image in which the source image and the processed image having the same field of view are arranged as the image elements on one screen. Further, the second mode is the processing mode for generating the output image in which a wide range image representing at least a partial region of the original image at a relatively low magnification and an enlarged image representing a partial region in the wide range image at a higher magnification than the wide range image are arranged as the image elements on one screen, the magnification of the enlarged image is changeable by the operation input, at least one of the wide range image and the enlarged image is the processed image and a region marker indicating a region corresponding to the enlarged image is superimposed on the wide range image.

Further, to achieve the above object, one aspect of an image processing system according to this invention includes an imaging device for generating the original image by imaging the pathological specimen, an image processing device having the above configuration, and a display device for displaying an image corresponding to the output image output by the image processing device.

In the invention, the “pathological finding” means knowledge obtained by applying an image analysis of extracting parts to be observed to the original image. For example, the “pathological findings” possibly include structures including tissues, cells, organelles and the like and liquids such as blood and interstitial fluid to be observed for a pathological diagnosis, the presence or absence of parts having a specific morphological feature in the original image, quantitative information such as the positions, sizes, number and the like of such parts and the like. In this sense, the pathological finding can be said to be substantially synonymous with “an analysis result obtained by the image analysis of extracting the parts to be observed”. Which parts are to be observed are determined according to the type of a disease and the purpose of the diagnosis.

Further, the “visual information” is various pieces of information which enables the user to easily visually recognize a specific region of the source image being the region extracted as the pathological finding and properties and the like of the pathological finding in this region. Modes of the visual information possibly include, for example, color coding, the emphasis of the contour or shading of the region, the attachment of markers by symbols or graphics or character information. These may be combined as appropriate.

In the inventions thus configured, the information on the pathological findings required by the user (doctor who diagnoses) can be easily presented according to a request and effectively support a diagnosis operation. The reason for that is as follows. At a pathological diagnosis site, a pathological specimen is observed from various perspectives and comprehensively diagnosed. Thus, a support method for displaying and presenting various pathological findings extracted from the pathological specimen in various display modes is more desirable than the presentation of only a quantitative evaluation result on a specific pathological finding.

In the first mode of the invention, using at least a part of an original image as a source image, an image including the source image and a processed image obtained by applying a processing based on finding information to the source image as image elements can be displayed. That is, the source image, which is an unprocessed image, and the processed image, which has the same field of view as the source image and to which visual information corresponding to pathological findings are added, are displayed on one screen. In such a display mode, the user can compare and observe the unprocessed source image and an extracted state of the pathological findings in the source image. This is, for example, useful in evaluating the positions, the distributed state, the density and the like of the pathological findings included in the source image.

In the processed image, a relative density between the source image and the visual information superimposed on the source image is changeable by a user operation. It is required to add clear visual information to clearly show the positions of the pathological findings, whereas it may become difficult to understand which structures in the image are regarded as the pathological findings due to the shielding of image contents of the source image by the visual information. By changing the relative density between the both according to the user operation, both the positions of the pathological findings and the image contents at these positions can be clearly shown in a mode desired by the user.

Note that a method for changing the density (luminance) of the source image, a method for changing the density of the visual information and a method for changing the both are available as a method for changing the relative density between the source image and the visual information. Any of these methods may be employed or these methods may be used while being switched as appropriate.

On the other hand, in the second mode of the invention, it is possible to display an image in which a wide range image representing a relatively wide region in the original image at a low magnification and an enlarged image representing a part of a region included in the wide range image at a high magnification are arranged on one screen. The second mode meets a demand in a pathological diagnosis to simultaneously grasp a state around a specimen while observing the details of the specimen. In this case, it is required in some cases to observe while a field of view of the enlarged image is increased or decreased in a stepwise manner. Accordingly, in the invention, a display magnification of the enlarged image can be changed by a user operation. Further, to show a correspondence relationship of the enlarged image and the wide range image, a region marker indicating the region corresponding to the enlarged image in the wide range image is displayed.

Such an image processing as to change a display magnification of an image according to a user operation is common. However, if an enlarged image having a changing display magnification and a wide range image displaying a wide range including the surrounding of the enlarged image at a fixed magnification are displayed on one screen and a region marker clearly showing a correspondence relationship of the both images is displayed, it is possible to meet a demand in a pathological diagnosis to simultaneously confirm a state of the surrounding while observing details at a suitable magnification.

In the image processing device of the invention, the processing mode desired by the user can be selected from those including these two processing modes and performed. Various methods for presenting an image of a specimen added with information useful for a pathological diagnosis have been proposed. However, only a specific display mode is not sufficient for the presentation of information required at a site of a pathological diagnosis comprehensively made from many pieces of information as described above. The invention has a plurality of processing modes covering various display modes required by the user and a variety of demands can be met by appropriately selecting and performing those processing modes.

As described above, the invention can be realized to cover various display modes required at a pathological diagnosis site. Further, in each display mode, an image can be variously changed according to a request of a user (doctor). Thus, various pieces of information obtained from the image can be presented in a mode easily understandable by the user and in a multifaceted way, and a diagnosis operation by the doctor can be effectively supported.

The above and further objects and novel features of the invention will more fully appear from the following detailed description when the same is read in connection with the accompanying drawing. It is to be expressly understood, however, that the drawing is for purpose of illustration only and is not intended as a definition of the limits of the invention.

