FIELD OF THE INVENTION

The invention relates to a method for producing and analyzing an overview contrast image of a sample carrier and/or of samples arranged on a sample carrier.

BACKGROUND OF THE INVENTION

Traditional light microscopes have been developed on the assumption that a user can look through an eyepiece onto a sample carrier while sitting or standing and interact directly with the sample in the sense that he will quickly be able to obtain an overview of the sample and of the field of view of the objective and laterally move the sample carrier with the sample either directly or using an adjustable sample stage so as to bring other regions of a sample into the field of view of the objective. The user of the microscope can remain at his location and need only minimally move his head, which means that in this respect, traditional microscopes are highly ergonomic.

Examination methods in particular for biological samples have been developed further over time, and as a consequence, the construction of microscopes that are suitable for performing said examination methods have also become ever more complex. In today's microscope systems, which permit the recording of image stacks along the observation direction and the reconstruction of a spatial image of the sample therefrom, images are produced with the aid of detectors. Detectors that are used are, for example, cameras that are configured with corresponding surface sensors, or photomultipliers. Consequently, the working space in such systems has shifted from the microscope stand and consequently from the sample to the computer or screen of such a computer. And yet, the working space in front of the stand is likewise used and required to prepare, i.e. set up, the sample for the examination. To this end, the sample on the sample carrier must be moved into the field of view of the objective, a sample region must be chosen, an adjustment to the position of said sample region must be performed, and, finally, the sample must be brought into focus. The workflow when using modern, complex microscope systems is therefore associated with two working spaces that represent different steps in the workflow and are spatially separate from one another—first, the microscope stand with the eyepiece for direct observation, and then the screen of a connected computer.

By attaching further instruments such as incubators for examining living cells, the direct view onto the sample, that is to say the position of the objective field of view in the sample, is greatly limited. If moreover larger sample carriers—for example multiwell plates—are used so that a multiplicity of samples can be examined in succession, the orientation on the sample is likewise obstructed.

As a result, finding the sample and setting the sample region are made more difficult for a user, and in addition, there is an orientation loss on the sample. Changing repeatedly between the computer working space and the microscope stand at which the sample can be directly observed is necessary for setting purposes.

In addition, microscope systems that are geared at the throughput of a high number of samples and are not continuously controlled also exist. In this case, sample regions must be automatically detected and microscopically captured. The sample carriers here typically carry an inscription, for example in the form of a barcode or a QR code, that must be assigned to the samples. Such examinations proceed semi-automatically; for example, a user only intervenes to change the sample carriers, to set an overview image, or to register the sample carrier number.

Especially when using multiwell plates in microscope systems with a high throughput, some of the wells may contain no samples or may contain incorrectly embedded, contaminated or defective samples. Said wells are likewise examined in the semi-automatic method, even though the results cannot be used, which means that the process takes more time than is actually necessary.

Although different methods exist in the prior art for producing overview images, these have more or less significant drawbacks. For example, an image can be recorded with the microscope optical unit and a weakly magnifying objective and a camera that is arranged downstream of the objective. However, this allows the recording of merely a small object field as compared to the size of the sample or of the sample carrier, in particular if sample carriers for a plurality of samples are used. In order to be able to record large sample regions, such as for example for multiwell sample carriers, it is therefore necessary to record a plurality of images of sections of the sample or of the sample carrier that are located next to one another and to subsequently combine them. This procedure is quite lengthy and not suitable for example for examining living samples.

An overview image can also be recorded if, instead of the microscope optical unit, a camera with a camera objective lens with which a relatively large object field can be imaged is used; the camera objective lens is generally not telecentric. This solution is implemented for example in the AxioScan series by the applicant, but can be used only with bright field illumination and incident light. Coverslips and undyed samples can be detected only with difficulty in this method.

Different solutions illuminate samples and sample carriers obliquely, i.e. at an angle that differs from zero relative to the optical axis, wherein the back-scattered light is detected. The sensitivity of the measurement over the object field to be measured here greatly fluctuates, which means that the results are not always reliable.

DESCRIPTION OF THE INVENTION

It is therefore an object of the invention to produce an overview image of a sample carrier in which the structure of the sample carrier itself and possibly further structures, for example of the inscription on the sample carrier, of possible sample regions, of any immersion liquid that may be present, of the actual sample and/or of sample errors such as air bubbles, dirt etc., can be detected clearly and without errors. Compared to normal recordings of a camera and even HDR images, the overview image is intended to have increased contrast or an improved signal-to-noise ratio of the particular structures of interest, which is why the overview image will be referred to as an overview contrast image below. The overview contrast image can be made available to the user for the purposes of navigation, or it can be used to improve the automated analysis of samples and to reduce the susceptibility to errors, for example by detecting sample errors.

This object is achieved by a method of the type described in the introductory part, in which the overview contrast image is produced as follows: A sample carrier—generally carrying at least one sample—is arranged at least partially in the focus of a detection optical unit and is illuminated in transmitted light with a two-dimensional, preferably array-type, illumination pattern. At least partially here means that the sample carrier or the sample does not need to be completely visible, but means in particular that the sample carrier and/or the samples can also have an extent along the optical axis that is greater than the depth of field of the detection optical unit used. A detection optical unit that can be used can be the optical unit of a microscope, although it is preferably a camera having a camera objective lens, which makes it possible to image a large object field onto a surface detector, preferably with a sufficiently high depth of field.

In this refinement of the method, it is necessary for producing the overview contrast image to detect at least two overview raw images with different illuminations of the sample carrier.

The overview raw images, that is to say unprocessed images, are then detected for example by the camera by way of a surface detector—for example a CMOS chip—registering the intensity pixel by pixel. Depending on the type of illumination, the overview raw images can be recorded successively or simultaneously, for example using a single camera.

A calculation algorithm that is used to calculate an overview contrast image from the at least two overview raw images is chosen in dependence on the type of the illumination and information that is to be extracted from the overview contrast image. The information can be, for example, the already mentioned structures of the sample carrier, of the sample etc., and of the inscription on the sample carrier.

Likewise in dependence on the information that is to be extracted from the overview contrast image, an image evaluation calculation algorithm is chosen that is used to extract the information from the overview contrast image. Said information can then be used for example by the user on the screen of a connected computer to initiate further steps as part of the observation and analysis, for example to navigate on the sample, which is accomplished by the image being represented on a screen and the user choosing the sample region of interest for example by way of a mouse click. On account of the image evaluation, the microscope system can then be automatically adjusted to that position. However, the information can also be used as part of an automated examination, for example in high-throughput examinations, to exclude incorrectly filled wells of a multiwell carrier, such as a microtiter plate, with the result that the microscope is not even adjusted to their positions.

Essential aspects relate to the use of a two-dimensional, in particular array-type illumination and to the recording of the overview raw images with different illuminations. Two-dimensional illumination can be obtained in different ways, wherein preferably an array of illumination elements of the same size is used to produce illumination patterns. At any rate, the individual illumination elements must be distinguishable from one another in the at least two overview raw images, that is to say they must be able to be represented separately from one another in the image, even though they were not arranged at the focus. The illumination elements can be LEDs, OLEDs, the ends of optical fibers, elements of an illuminated DMD (digital micromirror device) or of a different spatial light modulator. They can be elements that actively or passively emit light. The illumination elements can also be produced for example with a light source that emits light over a surface and upstream of which for example a switchable filter structure is arranged, with which one or more properties of the illumination elements—for example color, intensity or polarization—can be manipulated. With particular advantage, however, light-emitting diodes (LEDs) can be used, because these can be arranged in nested arrays of multicolor LEDs and also provide a sufficiently high light output, and moreover, microscopes that use an LED array, that is to say LEDs arranged in the manner of a matrix or a grid (LED array microscopes, angular illumination microscopes—AIM), for illumination are already available. The LED arrays of such microscopes can be likewise used to produce two-dimensional illumination patterns.

The different illuminations with which the at least two overview raw images are recorded can be realized in different ways. One simple possibility is to use a static two-dimensional illumination pattern and to move the sample carrier laterally, that is to say perpendicularly to the optical axis of the detection optical unit, relative to the illumination pattern between two recordings. This can be accomplished either by moving the illumination pattern itself—likewise arranged in a plane with the optical axis as a normal—or by moving the sample carrier. The sample carrier or the sample is here illuminated in transmitted light, that is to say the sample carrier is located between the illumination elements of the illumination pattern and the detection optical unit, for example the camera.

In addition to spatially different illuminations, it is also possible to use illuminations that differ in terms of time, for example by recording a plurality of overview raw images with different exposure times or with illumination of differing lengths with the same exposure time, with the signal-to-noise ratio being less favorable in the latter case. Using the HDR (high dynamic range) method known in the prior art, an overview contrast image can be calculated from said overview raw images.

Another possibility is to produce spatially different illuminations using different illumination patterns, wherein the illumination patterns are preferably chosen in dependence on the information that is to be extracted. In principle, a multiplicity of illumination patterns that can be impressed on the array of illumination elements, for example, are suitable. For example, different illumination patterns can be produced by driving the illumination elements individually or in groups and switching them to produce different illumination patterns, wherein a first portion of the illumination elements is switched to emit light and at least a second portion of the illumination elements is switched to emit no light or to emit light of a different color or light of a different polarization. If the at least second portion of the illumination elements does not emit light, each pattern includes exactly two parts, and the overview raw images are detected successively. If the second portion of the illumination elements emits lights of a different color, the illumination elements can also be divided into more than two groups and comprise a third or further parts that emit light of respectively different colors, wherein the colors differ from one another in pairs. A corresponding statement applies to the polarizations. Upon illumination with light of different colors, the overview raw images can be recorded simultaneously, provided that, on the detection side, separation into the different color channels is effected. For example, if the array of illumination elements comprises LEDs in the three RGB primary colors red (R), green (G) and blue (B), and if the sensor of the camera has corresponding sensors assigned to said color channels, a separation is readily possible, and three overview raw images can be recorded at the same time. Similar is true for polarized illumination, for example if an LED array is provided with polarization filters of different polarizations and the polarization directions are likewise detected and used for separating the channels.

If the illumination elements are in the form of LEDs, and an LED is formed from three individual mutually adjacent sub-LEDs that each emit light in different primary colors red, green and blue, the different illuminations can also be set by illumination from different angles in the primary colors. Here, too, the overview raw images can be detected simultaneously.

Another possibility is to produce explicitly different illumination patterns and to record the overview raw images successively with different illumination patterns. This can be done in a first variant for example by stochastically choosing the first portion of the illumination elements for each illumination pattern, wherein the individual illumination elements can randomly be driven and switched to emit light or to emit no light, wherein care should be taken to ensure by way of boundary conditions the best possible equal distribution of illumination elements that emit light and those that do not emit light. In a second variant, pulse-width-modulated illumination elements are used—this can be realized particularly well with LEDs or OLEDs—wherein the pulse width is selected to be longer than the integration time of a detector unit for recording the overview raw images. In this case, the illumination elements do not need to be driven individually.

Instead of stochastic illumination patterns, it is also possible to produce illumination patterns in which the illumination elements exhibit a regular distribution. For example, chessboard-type, cross-shaped or half pupil distributions can be used for the light-emitting illumination elements. In particular in the case of chessboard-type distribution, there are two good possibilities: First, the second portion of the illumination elements may emit no light, in which case two overview raw images are recorded successively with mutually complementary illumination patterns. If the illumination is in the manner of a chessboard, the patterns are inverted with respect to one another. Second, the two parts of the illumination elements can also emit light of different colors or polarizations, in which case the overview raw images can be recorded simultaneously in one image and subsequently be separated into color channels or polarizations. In the case of a chessboard-type illumination with two illumination patterns, the two patterns are not only complementary, but also inverted with respect to one another. If a plurality of patterns—for example a single light-emitting illumination element that scans the array—are used, all patterns together behave in a mutually complementary fashion, that is to say, overall, they produce an array of only light-emitting illumination elements. In the case of half pupil illumination, in each case two out of the four necessary illumination patterns are mutually complementary.

Finally, it is also possible to realize different illuminations by selecting from the array of illumination elements at least one section and producing the illumination pattern only in said section. The different illuminations are achieved by a scanning movement of the at least one section on the array, wherein the illumination elements outside of the at least one section are switched to emit no light. For example, illumination with an individual LED can be used here, or a section of a chessboard-type illumination. If a sample carrier is large, it is possible here to choose a plurality of sections that are moved in parallel fashion; a combination with differently colored illumination elements is likewise possible to produce a plurality of sections at the same time.

After the overview raw images have been recorded, a calculation algorithm that is used to calculate the overview contrast image from the at least two overview raw images is chosen in dependence on the information that is to be extracted from the overview contrast image. The choice of the calculation algorithm is preferably also made in dependence on the previous choice of the illumination method, i.e. is adapted thereto. The overview contrast image is preferably produced either in dark-field mode or in bright-field mode, because these permit the best contrast, but mixed modes are also possible. It is necessary here to ensure that the contrast is optimum for the structures of interest, for example depending on whether the cover slips, the sample itself, or the inscription is/are to be represented with the highest possible contrast. Under certain circumstances, overview contrast images can be produced from the overview raw images both in bright-field and in dark-field mode, depending on the desired information. In addition to the production of an overview contrast image in a dark-field or bright-field mode, other types of contrast can also be produced, for example overview contrast images in a HDR mode, which contain dark-field and bright-field components.

In a first configuration, the calculation algorithm is based on a pixel-wise projection, preferably a ranking projection or a projection of statistical moments. For producing the overview contrast image from a stack of at least two overview raw images, the overview raw images are compared pixel by pixel, and the intensity value of one of the pixels for the corresponding position in the overview contrast image is chosen in accordance with a projection condition. The recorded image stack is consequently subjected to calculation pixel by pixel, that is to say each pixel in the overview contrast image is influenced only by the pixels in the image stack that are located at the same image position in the overview raw images. In the case of a ranking projection, the corresponding pixel values of the overview raw images for an image position are sorted by intensities, and the value corresponding to the p-quantile is used for the overview contrast image, with p being a parameter that is to be specified by the user or is specified by the calculation method. A special case is the minimum projection with p=0.0, in which the pixel with minimum intensity is chosen; other special cases are the maximum projection with p=1, in which the pixel with maximum intensity is chosen, or the median projection with p=0.5.

Depending on the illuminations with which the overview raw images were produced, overview contrast images can be produced in dark-field contrast or in bright-field contrast with this type of calculation. An overview contrast image in bright-field mode can be produced for example if the illumination elements—in particular if these are in the form of LEDs—are not overdriven and p=1, i.e. the maximum projection is chosen. However, the overview contrast image can be also produced in dark-field mode, for example if the brightest possible overview contrast image is produced with the greatest possible p, wherein only pixels that have not been directly illuminated by an illumination element will be taken into account in the calculation. For example, if two overview raw images are produced with a chessboard-type distribution of the first portion of the illumination elements and a distribution that is complementary thereto, p=0 is selected and a minimum projection is performed. If, by contrast, the illumination pattern is produced only in a section of the array and this section is moved in a scanning fashion on the array, each pixel in the image is directly illuminated less often than overview raw images are recorded. For example, if four LEDs that are switched on in the form of a cross are used and 30 overview raw images are produced for a scanning chessboard pattern, each pixel is illuminated directly at most four times by an LED having a significantly larger diameter than a pixel. The value is then p=((30−1)−4)/(30−1)=0.8621.

Alternatively, it is also possible to use an algorithm based on the projection of statistical moments. In this case, each pixel in the overview contrast image corresponds to a statistical moment, such as for example the standard deviation of the intensity over the corresponding pixels of the overview raw images. In particular in combination with a sequence of statistical LED patterns that are moved laterally relative to the sample carrier, this offers good contrasts and maintains even small details, with the result that this calculation algorithm is particularly suitable for example for detecting multiwell sample carriers or chamber-slide sample carriers.

The advantages of the above-described projection methods as calculation algorithms are that they are able to be parallelized very well and consequently permit very fast calculation, and, in addition, owing to the equal treatment of all pixels, no seam artefacts occur, as would be the case for example with calculation algorithms based on segmentation, in which, in unfavorable cases, the boundaries of the structures seem to discontinuously jump at the seams.

In another configuration, the calculation algorithm is based on morphological operations with subsequent pixel-wise projection, preferably on a top-hat or black-hat transform with subsequent pixel-wise maximum projection. It is possible using the top-hat transform to highlight bright structures on a dark background, and the black-hat transform can be used to highlight dark structures on a bright background. These calculation algorithms can be used to make in particular glass edges, that is to say edges of sample carriers or cover slips, visible. Next, a pixel-wise maximum projection over the overview raw images thus transformed is formed and the overview contrast image is produced in this way. The advantages of this calculation algorithm are that the information is acquired at the same time from bright-field and dark-field contrasts, and that it is likewise efficiently subjected to a calculation. However, as compared to a ranking projection, the contrast in the images that have been subjected to calculation is mostly lower and is frequently visible only in the case of glass edges. In addition, strong background artefacts can be produced that must then be redressed.

In a further configuration, a calculation algorithm based on segmentation is chosen, in which initially a determination is made for each pixel of an overview raw image as to whether said pixel has been directly irradiated with light by an illumination element. These pixels are then not taken into account for the production of an overview contrast image in the dark-field contrast mode. The overview contrast image is produced using a projection method in this case, too. The advantage of this calculation algorithm as compared to the ranking projection is that an explicit determination is made here as to which pixel information from the overview raw images can be used. A disadvantage is that, on account of the segmentation and the resulting unequal treatment of pixels, seams can form in the calculated contrast image. In addition, the calculation cannot be performed as efficiently as in the previously described calculation algorithms.

In one alternative configuration of the method operating in bright-field mode, the overview contrast image is not produced by calculation but is detected directly, that is to say no overview raw images are detected, or, in other words, the overview raw image in this case is identical to the overview contrast image. To this end, a diffusion screen is inserted into the beam path between the array-type illumination pattern—which can be made up of an array of illumination elements of preferably equal size in this case, too—and the sample carrier. The diffusion screen, which produces diffuse illumination, which is advantageous for bright-field illumination, can also permanently remain in the beam path, provided it is switchable, and it is then switched on, i.e. switched to diffusion, only for the production of an overview contrast image in bright-field mode.

If the sample carrier is laterally moved relative to the illumination pattern between two recordings, it is necessary to know for the calculation how the sample or the illumination pattern in that case moves in the image. To this end, the camera is calibrated relative to the sample carrier or the illumination pattern in order to be able to map—in the case of a movement of the sample carrier—the coordinates of a stage, on which the sample carrier is held and which can be displaced to move the sample carrier, onto image coordinates. A similar procedure can be used for movable illumination patterns. In the case of a displaceable stage, a calibration pattern, for example a chessboard, is placed in the stage concretely for calibration purposes, as a result of which the coordinate mapping can be estimated with sufficient accuracy. Alternatively, such calibration can also be dispensed with in the calculation and the movement of the sample can be ascertained by way of image analysis or using a different measurement system.

In particular if the illumination pattern is stationary and the sample carrier is moved between the recordings, an overview contrast image can be generated after calibration, that is to say after a quantification of the actual movement of the sample carrier in relation to the image, even for larger sample carriers that do not entirely fit into the object field that is capturable by the camera or the detection optical unit, by initially producing individual contrast images that each show different regions of the sample carrier—and/or of the sample—and are produced from corresponding individual overview raw images. Said individual contrast images are subsequently combined to form the overview contrast image, wherein the calibration is used to correctly locate connecting points for joining them.

Notwithstanding the above, a calibration is also advantageously performed using the recording and evaluation of a calibration pattern for correcting geometric distortions before the recording of overview raw images. The calibration pattern is an object of known geometry and clearly detectable structure—for example the aforementioned chessboard pattern—which is placed at different positions in the image field of the camera and is recorded with the camera. However, it is also possible to use the array of illumination elements, in particular if these are in the form of LEDs, as a calibration pattern. Such calibrations are known from the prior art.

As has already been indicated in connection with calculation algorithms based on segmentation, background artefacts may occur in the produced overview contrast image depending on the calculation algorithm used. Such disturbing background signals are removed, i.e. by calculation, preferably after the production of the overview contrast image yet before the evaluation thereof using corresponding correction algorithms. If there is no lateral movement of the illumination pattern or the sample carrier, the artefacts typically form a periodic structure that follows the positions of the individual illumination elements. These can then be removed using what is known as self-filtering. Additional correction methods are known in the prior art that can be used to remove the occurring background artefacts or at least to reduce them, in particular by recording or calculating a background image, which is then subtracted from the overview contrast image, i.e. removed therefrom by calculation. For example, a background image can be determined from the calculation of the overview raw images by averaging out the foreground. A background image can also be determined from a recording without the sample carrier or with an empty sample carrier. By subjecting the background contrast image to a calculation, a background image can be determined for example by calculating the average pixel values in local regions around an illumination element that does not emit light by way of analyzing all illumination elements in the overview contrast image. This is because the foreground structure is independent of the position relative to the illumination element and is averaged out, but the background structure is dependent on said position and is consequently intensified. This produces a background image via the position of the illumination elements that can subsequently be calculated out of the overview contrast image. Another possibility for calculating out background artefacts is the use of a bandpass filter, possibly also in combination with non-linear filters.

In a final step, the background contrast images are finally automatically analyzed using an image processing algorithm that is to be chosen and the required information is extracted. The information that is to be extracted comprises one or more of the following data: type of the sample carrier, inscription in the sample carrier, locations of samples or sample regions, cover slips, wells of a multiwell plate in the image, information relating to an immersion liquid such as position, volume, shape, positions of artefacts, defective samples, air bubbles etc. This information can be reliably extracted only on account of the high contrast in the overview contrast images.

An image processing algorithm that can be chosen is for example an algorithm based on the principle of machine learning, in particular a deep learning algorithm, which is preferably trained on the basis of overview contrast images with known information.

For automated extraction of the aforementioned information from the overview contrast images, it is expedient to use methods from the field of machine learning. In this case, an annotated training sample comprising a quantity of contrast images that are to be analyzed is provided for the respective task, including for example image classification, segmentation, localization, detection. Each contrast image is here assigned a desirable output corresponding to the task, as will also be explained below with reference to examples. Using techniques of machine learning, it is then possible for a model to be automatically adapted in a learning step such that the desirable and correct outputs are produced even for non-viewed, i.e. new images.

Possible algorithms based on techniques of machine learning will be outlined by way of example below. Alternatively, methods from traditional image and signal processing can be used, but algorithms based on machine learning, and in particular based on deep learning, offer significant advantages for example in terms of quality, robustness, flexibility, generalizability, and development and maintenance complexity.

A deep learning algorithm based on a convolutional neural network can be advantageously used to identify the type of a sample carrier, for example whether it is for example a multiwell sample carrier, a Petri dish, a chamber slide etc. The task is thus one from the field of image classification, and the algorithm uses an image as input and outputs a class. The training sample in this case comprises contrast images, and each contrast image is assigned one of the suitable sample carrier types, a first contrast image is assigned the type “Petri dish,” a second contrast image is assigned the type “multiwell plate,” a third contrast image is assigned the type “chamber slide,” a fourth contrast image is assigned the type “slide,” and so on.

A convolutional neural network (CNN) consists of different layers, for example convolutional layers, pooling layers, non-linear layers, etc., the arrangement of which is specified in the network architecture. The architecture used for the image classification follows a certain basic structure, but is in principle flexible. Each element of the network receives an input and calculates an output. In addition, some elements of the network have free parameters that determine the calculation of the output from the input. A three-dimensional number array, i.e. a color image having in each case three color values per pixel, is input to the first layer as an input of the network. The last layer then outputs a probability distribution over all possible sample carrier types as the output of the network—for example the output for an overview contrast image is thus: “slide” 87%, “multiwell plate” 1%, “Petri dish” 2%, “chamber slide” 10%. Optionally, a rejection class—which provides for example the values “unknown”/“invalid”/“empty”—can also be integrated. On account of the training process, the free parameters of the network are adapted on the basis of the provided training data such that the outputs of the model match the expected outputs as closely as possible.

The training can also use a model that has already been trained for other data as a starting point in the sense of a fine adjustment, which offers advantages in terms of quality, time requirements, and data expenditure.

As an alternative to CNNs or methods derived therefrom or related methods of deep learning, the image classification can also be performed using traditional methods of machine learning, which typically comprise two steps: (I) feature extraction, and (ii) classification. In the case of the feature extraction of step (i), the overview contrast image is transformed, using a predefined algorithm, into an alternative representation—typically into a compact or sparse vector. Simple examples here are for example local gradient histograms (histograms of oriented gradients, HoG). In the classification of step (ii), each of these feature vectors is then assigned a class using a classifier. One example of a classifier is a support vector machine (SVM). The free parameters of the classifier are here likewise adapted in the training step such that the actual outputs match the desirable outputs as closely as possible.

A hybrid approach between traditional machine learning and deep learning is based on the use of a CNN for the feature extraction in step (i). In this case, a CNN that has been trained for different data is cut off at a specific layer, and the activation of the network is used as a feature vector.

The localization of structures of the sample carrier, for example cover slips in the case of slides and Petri dishes or chambers of chamber slides or multiwell plates, on the one hand, and of structures of the sample or sample regions on the other can be considered a problem of semantic segmentation, that is to say, for an overview contrast image as the input image, an image in which each pixel of the input image is assigned a class (e.g. “background,” “cover slip,” “dirt,” “sample,” . . . ) is to be returned. This can preferably likewise be implemented with networks from the field deep learning, for example using fully convolutional networks (FCNs) that are based on CNNs.

Like CNNs, FCNs typically expect as an input a three-dimensional number array, i.e. a colored overview contrast image, but output an array in which each pixel of the input image is assigned a probability relating to it being part of each of the occurring classes. The training sample in this case comprises contrast images, and each contrast image is assigned an array (graylevel image) of the same size, in which each pixel is in turn assigned a class—coded via a gray level. Training proceeds similar to in the case of CNNs. It is possible here to use different FCNs for the localization of sample carriers and for the localization of sample regions, but the evaluation is in particular possible using a single FCN, which images or contains for example both “coverslip” and “sample” as classes.

It goes without saying that the aforementioned features and those yet to be explained below can be used not only in the combinations specified but also in other combinations or on their own, without departing from the scope of the present invention.

