This application is the U.S. National Phase application under 35 U.S.C. § 371 of International Application No. PCT/EP2018/082474, filed on 26 Nov. 2018, which claims the benefit of European Application Serial No. 17204870.4, filed 1 Dec. 2017. These applications are hereby incorporated by reference herein.

FIELD OF THE INVENTION

The invention relates to a segmentation system, method and computer program for segmenting an object in an image. The invention further relates to a training system, method and computer program for training a neural network.

BACKGROUND OF THE INVENTION

The article “Segmentation of the heart and great vessels in CT images using a model-based adaptation framework” by O. Ecabert et al., Medical Image Analysis, volume 15, pages 863-876 (2011) discloses a model-based segmentation technique for segmenting a heart in an image. The model-based segmentation includes placing a surface model of the heart in the image, wherein the surface model comprises surface elements being triangles. For each surface element candidate points are sampled along a line perpendicular to the respective surface element and for each of these candidate points the likelihood is determined that it corresponds to a boundary of the heart, wherein then the candidate point with the highest likelihood is selected and the position of the respective surface element is adapted accordingly. For the selection of the candidate point with the highest likelihood triangle-specific acceptance criteria can be used like the criteria disclosed in the article “Optimizing boundary detection via Simulated Search with applications to multi-modal heart segmentation” by J. Peters et al., Medical Image Analysis, volume 14, pages 70-84 (2010). This model-based segmentation does not always provide good segmentation results.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a segmentation system, method and computer program which allow for an improved segmentation of an object in an image. It is a further object of the present invention to provide a training system, method and computer program for training a neural network, which can be used for providing the improved segmentation.

In a first aspect of the present invention a segmentation system for segmenting an object in an image is presented, wherein the segmentation system comprises:an image providing unit for providing an image of an object, the image representing an image volume,a model providing unit for providing a deformable surface model for being adapted to a surface of the object, wherein the surface model comprises surface elements defining respective parts of a mesh surface,a subvolumes determination unit for placing the surface model within the image and for determining for each surface element of the surface model a respective subvolume of the image such that the respective subvolume overlaps with the respective surface element,a neural network providing unit for providing a convolutional neural network being adapted to determine distances between surface elements of the surface model and a boundary of the object in the image based on determined subvolumes,a distance determination unit for determining respective distances between the surface elements of the provided and placed surface model and the boundary of the object in the provided image by using the provided neural network based on the determined subvolumes, anda model adaptation unit for adapting the surface model to the object in the image in accordance with the determined distances.

By determining for a respective surface element a respective subvolume, by using the subvolumes together with the convolutional neural network for determining the distances between the surface elements and the boundary of the object in the image and by then using these distances for adapting the surface model, the segmentation of the object in the image is improved in comparison to known techniques which rely, for instance, on a sampling of candidate points along lines being perpendicular to the respective surface element and on a determination of likelihoods for the candidate points that they indicate a boundary of the object.

The image providing unit can be a storing unit, in which the image of the object is stored, and from which the image can be retrieved for providing the same. The image providing unit can also be a receiving unit for receiving an image of the object and for providing the received image. Moreover, the image providing unit can be an image generation unit being adapted to determine the image based on raw data of the object. The image providing unit can also be an entire imaging system being adapted to acquire the raw data and determine, in particular, reconstruct, the image based on the raw data.

In an embodiment the neural network providing unit is adapted to provide a further convolutional neural network being adapted to determine confidence values for surface elements of the surface model based on the subvolumes, wherein a confidence value for a respective surface element is indicative of an estimation of a deviation of the distance determined for the respective surface element from the actual distance of the respective surface element to the boundary of the object in the image, wherein the segmentation system further comprises a confidence value determination unit for determining confidence values for the surface elements of the surface model by using the provided further neural network based on the determined subvolumes. In particular, the model adaptation unit is adapted to adapt the surface model in accordance with the determined distances and based on the respective confidence value determined for the respective surface element. For instance, the model adaptation unit can be adapted to allow a distance determined for a respective surface element, for which a relatively large confidence value has been determined, to contribute stronger to the adaptation process than a distance determined for a respective surface element for which a relatively low confidence value has been determined. Accordingly, the degree of contribution to the adaptation process of a distance determined for a respective surface element may depend on the confidence value determined for the respective surface element. This can lead to a further improved segmentation of the object in the image.

In an embodiment the neural network providing unit comprises several convolutional neural networks which are adapted to be used for different surface models representing different kinds of objects and/or for different groups of surface elements of a same surface model, wherein the neural network providing unit is adapted to provide, for determining distances for all surface elements of a surface model representing the object or for a group of surface elements of the surface model representing the object, the corresponding neural network. Thus, for different parts of a same object, different neural networks might be used for segmenting the object. Moreover, for segmenting the respective object, a neural network might be used, which is specifically adapted for the class of objects, to which the object to be segmented belongs to. This can lead to a further improved determination of the distances between the respective surface elements and the boundary of the object in the image and hence finally to a further improved segmentation of the object in the image.

The model providing unit is preferentially adapted to provide the surface model such that each surface element comprises a direction, wherein the distance determination unit is adapted to determine the respective distance in the direction of the respective surface element. Moreover, preferentially the model providing unit is adapted to provide a triangle mesh as the surface model, wherein the surface elements are triangles. However, the surface model can also be another mesh, wherein the surface elements can have another shape not being triangular. The object is preferentially an anatomical object like an organ or another part of a human being or of an animal. However, the object can also be a technical object.

In particular, the model providing unit provides the surface model such that for each surface element a respective three-dimensional orthogonal coordinate system is defined, wherein one axis of the coordinate system coincides with the normal of the respective surface element. The direction of the axis, which is aligned with the normal, can be arbitrarily defined and defines the direction of the respective surface element. For instance, if the surface element is a triangle having a first vertex, a second vertex and a third vertex, the direction of the axis, i.e. the orientation, might be defined by the cross product of a) a vector pointing to the second vertex minus a vector pointing to the first vertex and b) a vector pointing to the third vertex minus a vector pointing to the first vertex.

The surface elements are preferentially planar and the two further axes are preferentially located in the respective plane of the respective surface element. For instance, the center of the coordinate system can coincide with the center of the respective surface element, wherein the first axis is aligned with the normal, the second axis is within the plane of the respective surface element and points from the center to one vertex of the surface element and the third axis can be orthogonal to the first and second axes.

The subvolumes determination unit is preferentially adapted to determine the subvolumes such that they are defined with respect to the coordinate system of the respective surface element. For instance, the respective subvolume can be rectangular having three axes, wherein one of these axes can coincide with the first axis of the coordinate system of the respective surface element being aligned with the normal of the respective surface element. Or, in another example, the respective subvolume can be cylindrical and the longitudinal axis of the cylindrical subvolume can coincide with the first axis of the coordinate system of the respective surface element. Since the respective subvolume is preferentially defined with respect to the coordinate system of the respective surface element, the subvolumes are preferentially independent of the pose and position of the object in the image. Correspondingly, also the input to the neural network is preferentially independent of the pose and position of the object in the image, thereby simplifying the neural-network-based segmentation of the object in the image.

The distance determination unit is preferentially adapted to determine signed distances, wherein the sign is a sign with respect to the axis of the coordinate system of the respective surface element, which is aligned with the normal of the respective surface element. The sign defines in which direction the boundary of the object is present in the provided image with respect to the surface element, i.e., for instance, whether the boundary is “above” or “below” the respective surface element, or, in other words, in which direction the respective surface element needs to be moved by the model adaptation unit, in order to adapt the surface model to the boundary of the object in the provided image. For instance, it can be defined that, if the respective sign is positive, the respective surface element needs to be moved in the direction of the axis of the coordinate system being normal to the respective surface element and, if the respective sign is negative, the respective surface element needs to be moved in a direction being opposite to a direction of the axis of the coordinate system being normal to the respective surface element.

In an embodiment the subvolumes determination unit is adapted to determine the subvolumes such that they are elongated. Furthermore, the model providing unit can be adapted to provide the surface model such that each surface element comprises, as explained above, a direction, wherein the subvolumes determination unit is adapted to determine the subvolumes such that they are elongated in the direction of the respective surface element. In particular, the corresponding elongation direction is preferentially aligned with the first axis of the coordinate system of the respective surface element being aligned with its normal. However, in another embodiment the subvolumes can also be not non-elongated. For instance, they can be cubical or spherical. Moreover, it is preferred that the subvolumes determination unit is adapted to determine the subvolumes such that they have the same dimensions and the same shape. By using the same dimensions and the same shape the computational efforts can be reduced.

In an embodiment the neural network providing unit is adapted to provide the convolutional neural network such that it is adapted to additionally determine further quantities being related to the image of the object based on the subvolumes of the image, wherein the distance determination unit is also adapted to determine the further quantities by using the provided neural network based on the determined subvolumes. In particular, the neural network providing unit and the distance determination unit can be adapted such that the further quantities include the normals of the boundary of the object in the image.

The model adaptation unit is adapted to adapt the surface model by considering the determined distances and optionally normals of the boundary of the object in the image. In particular, the model adaptation unit can be adapted to use an adaptation algorithm having a term in its cost function which tries to attract the respective surface element to a respective target point defined by the determined distance. For instance, the respective target point can be defined by the respective distance and the respective axis of the coordinate system being aligned with the normal of the respective surface element, wherein the target point is the point on the respective axis, which has the respective distance to the respective surface element. In a further embodiment the model adaptation unit is adapted to use a cost function with a term that tries to attract the respective surface element not to the target point, but to a target plane approximating a tangent plane, wherein the target plane is preferentially defined by the corresponding normal of the boundary of the object, i.e. the normal is perpendicular to the target plane, and the respective distance. This attraction to the target plane and not to the target point allows the respective surface element to “slide” on the boundary of the object, which can lead to an improved adaptation of the surface model to the object in the image. In general, the model adaptation unit is preferentially adapted to use a cost function having different terms, wherein the adaptation process is carried out by minimizing the cost function. The cost function comprises a term which depends on the determined distances, wherein this term is reduced, if the distance between the surface elements and the boundaries are reduced. This term tries to, for instance, attract the respective surface element to a target point defined by the respective determined distance or to a target plane defined by the respective determined distance and normal. Furthermore, the model adaptation unit can be adapted to use an adaptation algorithm having a term in its cost function which tries to orient the respective surface element such that it is parallel to the target plane approximating the tangent plane. The cost function can comprise further terms like a term which tries to keep a certain shape of the surface model. The terms of the cost function might also be regarded as being “energies”, i.e. one of these terms might be regarded as being an internal energy and another of these term might be regarded as being an external energy. For more details regarding the adaptation of the surface model reference is made to known adaptation algorithms like the algorithms disclosed in the above mentioned articles by O. Ecabert et al. and J. Peters et al.

In an embodiment the neural network providing unit is adapted to provide a single convolutional neural network being adapted to determine distances to the boundary of the object for all surface elements of the surface model, wherein the distance determination unit is adapted to determine the respective distances between the surface elements of the surface model and the boundary of the object in the image by using the provided single neural network based on the determined subvolumes. Moreover, in an embodiment the image providing unit is adapted to provide the image such that each image element comprises two or more image values. In particular, the image providing unit can be adapted to provide the image such that the image values of a same image element correspond to different imaging modalities or to different used image acquisition protocols of a same imaging modality. If a single image element, i.e., for instance, a single voxel, comprises two or more image values, the information of each image element is increased in comparison to having only a single image value for a single image element. This increased information provided by the image can lead to a further improved determination of the distances between the surface elements of the surface model and the boundary of the object in the image, particularly if the different image values of a same image element correspond to different imaging modalities which often emphasize different aspects of a same object. In the case of providing an image, wherein a single image element has several image values, the neural network preferentially has also been trained with images having several image values in a single image element.

The subvolumes determination unit is preferentially adapted to determine the subvolumes by sampling the image. In an embodiment the image is sampled such that a degree of sampling depends on a distance from a center of the respective subvolume. In particular, the image is sampled such that a degree of sampling decreases with increasing distance from a center of the respective subvolume. For instance, if the subvolumes are rectangular, the image can be sampled such that a degree of sampling decreases with increasing a distance from the normal of the respective surface element at the center of the respective subvolume in a direction being parallel to the respective surface element.

The subvolumes are preferentially rectangular. However, in an embodiment they can be cylindrically shaped. In particular, the subvolumes determination unit can be adapted to determine the subvolumes by sampling the image such that a degree of sampling along a ring shaped part of the cylinder depends on the radius of the ring. For instance, the sampling rate can be reduced with increasing distance to the center of the respective surface element, wherein the resulting reduced overall sampling rate can lead to reduced computational efforts needed for segmenting the object in the image. The reduction of the sampling rate corresponds to an increase of gaps between samples. In an embodiment a fixed number of samples is used for all rings such that with increasing diameter naturally also the gap between the samples is increased, where the increase in gap directly relates to the distance of a sample from the center line of the cylinder. Alternatively, if the subvolumes are rectangular, rectangles, particularly squares, of increasing size in a plane being perpendicular to the respective surface element might be used while always sampling the same amount of sample points. Preferentially, for different surface elements the same sampling pattern is used for determining the different subvolumes, wherein this sampling pattern might be a hexagonal pattern or another pattern. If the subvolumes are cylindrical, the sampling pattern might be defined by two or more rings, wherein the samples on neighboring rings are shifted with respect to each other in a direction being orthogonal to the radius.

In an embodiment the segmentation system further comprises a training data providing unit for providing a training image showing a training object and for providing a deformable training surface model which comprises several surface elements and which has been adapted to the training object, and a training unit for a) determining several modified training surface models by modifying surface elements of the adapted training surface model, b) determining subvolumes of the training image for the surface elements of the modified training surface models, wherein for a respective surface element a subvolume is determined, which overlaps the respective surface element, c) determining distances for the surface elements of the modified training surface models, wherein for a respective surface element a respective distance to the un-modified training surface model, which has been adapted to the training object in the training image, is determined, and d) training the provided convolutional neural network based on determined subvolumes and determined distances. Thus, the segmentation system cannot only be adapted to segment the object, but also to train a new or an already trained convolutional neural network for the segmentation procedure.

In a further aspect of the present invention a training system for training a neural network is presented, wherein the training system comprises:a neural network providing unit for providing a convolutional neural network,a training data providing unit for providing a training image showing a training object and for providing a deformable training surface model which comprises several surface elements and which has been adapted to the training object,a training unit for training the provided neural network, wherein the training unit is adapted to:

a) determine several modified training surface models by modifying surface elements of the adapted training surface model,

b) determine subvolumes of the training image for the surface elements of the modified training surface models, wherein for a respective surface element a subvolume is determined, which overlaps the respective surface element,

c) determine actual distances for the surface elements of the modified training surface models, wherein for a respective surface element a respective distance to the un-modified training surface model, which has been adapted to the training object in the training image, is determined,

d) train the provided convolutional neural network based on the determined subvolumes and the determined actual distances.

The training data providing unit is preferentially a storing unit in which the training image and the adapted training surface model are stored and from which the training image and the adapted training surface model can be retrieved for providing the same. However, the training data providing unit can also be a receiving unit for receiving the training image and the adapted training surface model from another unit and for providing the received training image and adapted training surface model.

Preferentially the training unit is adapted to displace a surface element and/or tilt a surface element for modifying the surface element. In this way several modified training surface models, which can be used for the training, can be determined in a relatively simple way. In particular, an unlimited amount of modified training surface models having known distances to the boundary of the training object can be generated for the training procedure as ground truth, in order to train the neural network as extensive as desired. This can lead to a very well trained neural network and hence to a very exact segmentation, if this neural network is used for segmenting an object in an image.

In an embodiment the training unit is adapted to a) determine simulated distances for the surface elements of the modified training surface models based on the determined corresponding subvolumes and the trained convolutional neural network and b) determine deviation values for the surface elements of the modified training surface models, wherein for a respective surface element a respective deviation value is determined, which is indicative of a deviation of the respective simulated distance from the respective actual distance, wherein the neural network providing unit is adapted to provide for surface elements, for which the respective deviation value is larger than a threshold, a further convolutional neural network, wherein the training unit is adapted to train the provided further convolutional neural network based on the subvolumes and the actual distances determined for the surface elements for which the respective deviation value is larger than the threshold. Thus, for different parts of the same object different neural network can be trained, wherein the respective neural network is optimized for the respective part of the object. If the correspondingly trained neural networks are used for segmenting an object in an image, the segmentation can be further improved.

In an embodiment the training unit is adapted to a) determine simulated distances for the surface elements of the modified training surface models based on the determined corresponding subvolumes and the trained convolutional neural network, and b) determine confidence values for the surface elements of the modified training surface models, wherein for a respective surface element a respective confidence value is determined based on a deviation of the respective simulated distance from the respective actual distance, wherein the neural network providing unit is adapted to provide a further convolutional neural network for determining confidence values for surface elements of a surface model of an object based on the subvolumes, wherein the training unit is adapted to train the further convolutional neural network based on the confidence values and the subvolumes of the training image determined for the surface elements of the training surface model. Thus, the training can also result in a confidence neural network, which provides confidence values during a segmentation process, wherein these confidence values can be used for further improving an adaptation of a model surface to an object in an image and hence can lead to a further improved segmentation. The confidence values preferentially depend on the deviations such that the larger the deviation the smaller the confidence values.

In another aspect of the present invention a segmentation method for segmenting an object in an image is presented, wherein the segmentation method comprises:providing an image of an object by an image providing unit, the image representing an image volume,providing a deformable surface model for being adapted to a surface of the object, wherein the surface model comprises several surface elements, by a model providing unit,placing the surface model within the image and determining for each surface element a respective subvolume of the image such that the respective subvolume overlaps with the respective surface element by a subvolumes determination unit,providing a convolutional neural network being adapted to determine distances between surface elements of the surface model and a boundary of an object in an image based on determined subvolumes by a neural network providing unit,determining respective distances between the surface elements of the provided and placed surface model and a boundary of the object in the provided image by using the provided neural network based on the determined subvolumes by a distance determination unit, andadapting the surface model in accordance with the determined distances by a model adaptation unit.

In a further aspect of the present invention a training method for training a neural network is presented, wherein the training method comprises:providing a convolutional neural network by a neural network providing unit,providing a training image showing a training object and providing a deformable training surface model which, comprises several surface elements and which has been adapted to the training object, by a training data providing unit,training the provided neural network by a training unit, wherein the training includes:

a) determining several modified training surface models by modifying surface elements of the adapted training surface model,

b) determining subvolumes of the training image for the surface elements of the modified training surface models, wherein for a respective surface element a subvolume is determined, which overlaps the respective surface element,

c) determining actual distances for the surface elements of the modified training surface models, wherein for a respective surface element a respective distance to the un-modified training surface model, which has been adapted to the training object in the training image, is determined,

d) training the provided convolutional neural network based on the determined subvolumes and the determined actual distances.

In a further aspect of the present invention a segmentation computer program for segmenting an object in an image is presented, wherein the computer program comprises program code means for causing a segmentation system as defined in claim1to carry out the steps of the segmentation method as defined in claim12, when the computer program is run on a computer controlling the segmentation system.

In another aspect of the present invention a training computer program for training a neural network is presented, wherein the computer program comprises program code means for causing a training system as defined in claim10to carry out the steps of the training method as defined in claim13, when the computer program is run on a computer controlling the training system.

It shall be understood that the segmentation system of claim1, the training system of claim10, the segmentation method of claim12, the training method of claim13, the segmentation computer program of claim14and the training computer program of claim15have similar and/or identical preferred embodiments, in particular, as defined in the dependent claims.

It shall be understood that a preferred embodiment of the present invention can also be any combination of the dependent claims or above embodiments with the respective independent claim.

These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

