CROSS-REFERENCE TO PRIOR APPLICATIONS

This application is the U.S. National Phase application under 35 U.S.C. § 371 of International Application No. PCT/EP2018/063431, filed on May 23, 2018, which claims the benefit of European Patent Application No. 17305614.4, filed on May 24, 2017. These applications are hereby incorporated by reference herein.

FIELD OF THE INVENTION

The present invention relates to a device for providing spatial information of an interventional device in a live 2D X-ray image, to an X-ray imaging system, to a method for providing spatial information of an interventional device in a live 2D X-ray image, to a computer program element for controlling such a device and to a computer readable medium having stored the program element.

BACKGROUND OF THE INVENTION

In X-ray guided interventions, fluoroscopy may be used to navigate a tool towards its target during intervention. Fluoroscopy is an imaging technique that uses X-rays to be able to obtain real-time moving images of an object. In particular, inserted interventional devices that are not visible by human eyesight are made visible in relation to the object.

For achieving depth information, additional X-ray images are required, such as bi-plane imaging. However, this may be cumbersome and may result in additional X-ray radiation.

Alternatively, a second imaging modality may be used, such as 3D ultrasound imaging. WO 2011/070477 A1 describes a combined X-ray and ultrasound imaging system, in which 3D ultrasound imaging data may, in real time, supplement the X-ray images. In addition to providing depth information, the ultrasound data may render visible structures with low X-ray visibility, such as soft tissue.

SUMMARY OF THE INVENTION

There may thus be a need to provide a facilitated visualization technique to provide 3D information of a subject.

The object of the present invention is solved by the subject-matter of the independent claims; further embodiments are incorporated in the dependent claims. It should be noted that the following described aspects of the invention apply also for the device for providing spatial information of an interventional device in a live 2D X-ray image, for the X-ray imaging system, for the method for providing spatial information of an interventional device in a live 2D X-ray image, for the computer program element for controlling such a device and also for the computer readable medium having stored the program element.

According to a first aspect, a device for providing spatial information of an interventional device in a live 2D X-ray image is provided. The device comprises an input unit and a processing unit. The input unit is configured to provide a live 2D X-ray image of a region of interest related to a portion of the patient's body. A target location lies within the region of interest. The input unit is further configured to provide live 3D ultrasound data of at least a part of the region of interest and at least a portion of an interventional device. The processing unit is configured to register the live 3D ultrasound data to the live 2D X-ray image. The processing unit is configured to identify the target location in the 3D ultrasound data, and to determine, based on the live 3D ultrasound data, a spatial relation between the portion of the interventional device and the target location. The processing unit is configured to generate a graphical representation indicative of the spatial relation. The graphical representation is provided to be displayed with the live 2D X-ray image.

As an exemplary embodiment of this first aspect, a device for providing spatial information of an interventional device in a live 2D X-ray image is provided. The device comprises an input unit and a processing unit. The input unit is configured to provide an actual (or live, or current) 2D X-ray image of a region of interest related to a portion of the patient's body. A target location lies within the region of interest. At least a portion of an interventional device is arranged in the region of interest. The input unit is further configured to provide actual (or live, or current) 3D ultrasound data of at least the part of the region of interest and at least the portion of the interventional device. The target location lies within the region of interest. The processing unit is configured to register the actual 3D ultrasound data to the actual 2D X-ray image. The processing unit is configured to identify the target location in the 3D ultrasound data, and to determine, based on the actual 3D ultrasound data, a spatial relation between the portion of the interventional device and the target location. The processing unit is also configured to retrieve a distance value in a projection direction of the actual 2D X-ray image from the spatial relation. The processing unit is configured to generate, based on the distance value, a graphical representation indicative of the spatial relation. The graphical representation is provided to be displayed with the actual 2D X-ray image. The graphical representation is thus displayable with the actual 2D X-ray image.

The term “patient” refer to a subject that is undergoing e.g. an interventional procedure or examination. The patient could also be referred to as “subject” or “individual”. The individual, i.e. subject, is thus the object of interest for the procedure.

In an example, the input unit provides the actual 2D X-ray image and the actual 3D ultrasound data.

The term “arranged in” refers to the device being present in the region of interest at least with a portion. The device is thus located or provided in the region of interest. In an example of the device being a catheter, guide wire, needle or the like, the device is inserted such that e.g. the tip, or any other predetermined portion, of the device is within the region of interest, i.e. in the field of view covered by the 2D and the 3D image data, i.e. the 2D X-ray image and the 3D ultrasound data.

As, in an example, the 2D X-ray image and the 3D ultrasound data are referring to the same actual point or instance in time, or time range or time-period, the respective 2D and 3D data shows the same region of interest, just with different data information due to different image data acquisition techniques. Hence, the interventional device being arranged e.g. with a tip portion in the region of interest will be present in both 2D and 3D data.

In an example, the region of interest is a portion of a patient's body, in which a user is navigating the interventional device to the target location. A predetermined point of the interventional device is to be navigated to a target location. For example, a tip portion is to be navigated to the target location. In a further example, a shaft or side portion is to be navigated to the target location.

In an example, the region of interest (as provided by the actual X-ray image) shows the predetermined point of the interventional device, i.e. the interventional device is moved such that the predetermined point of the interventional device is arranged within the region of interest.

In an example, the processing unit is configured to provide image data for displaying the actual 2D X-ray image and the graphical representation.

In an example, the portion of the interventional device relates to a part of the interventional device with a point or portion that is to be navigated to the target. The portion of the interventional device thus comprises a predetermined point of the interventional device.

In an example, the 3D ultrasound data comprises 3D ultrasound data of the predetermined point of the interventional device, i.e. the interventional device is moved such that the predetermined point of the interventional device is arranged within the part that is covered by the ultrasound data.

The target location is also referred to as a predetermined location defined as target or target position. In an example, the user is navigating a predetermined point of the interventional device to the predetermined target location.

The term, “indicative” refers to a representation that allows the user to perceive the actual distance in a preferably facilitated manner.

In an example, the graphical representation is based on the spatial relation.

In an example, the graphical representation is displayed with the actual 2D X-ray image.

In an example, the term “to identify” relates to determining the target location in the 3D ultrasound data. This may be provided manually by the user. In another example, this is provided by

The term “actual” refers to a current situation, in which the 2D X-ray image and the 3D ultrasound data are acquired.

The term “live” relates to actual (or current) image data. Live, i.e. actual (current), means that any change, i.e. movement of the interventional device is shown instantly on the 2D X-ray image.

The term “navigating” relates to handling the interventional device by a user in a way to arrive at the target location.

In an example, the registration of the current 2D X-ray image and the 3D ultrasound data is provided by registering their respective spatial frames. In an example, the respective image acquisition means are registered to each other. In another example, the registration of the current 2D X-ray image and the 3D ultrasound data is provided by landmarks visible in both image types. In a further option, the registration is provided by the interventional device.

The term “target location” is related to a particular point of interest inside the object. The particular point of interest can be static in the relation of the object, i.e. not moving in relation to the object. In another example, the particular point of interest can be moving with the object, e.g. due to patient motion. In a further example, the particular point of interest can be moving in relation to the object, i.e. moving within the object.

The graphical representation can also be referred as to distance coding. This means that the graphical representation is designed to indicate different depth levels of portion of interventional device in relation to the target location. For example, different colors can be defined for a code pattern.

The 2D X-ray image can also be referred to as projection image of the patient's body in an x-y plane.

The actual 3D ultrasound data can also be referred to as actual 3D ultrasound image data.

The 3D ultrasound data can be related to image data which includes additional spatial information of the acquired image in the z-direction, in addition to spatial information in the x-y plane.

The spatial relation of the point of the interventional device and the targeted location can also be referred to as distance values in x-, y and z-direction.

The distance value in the projection direction of the actual 2D X-ray image could also be referred to as depth or depth value or as a distance value in z-direction of the 2D X-ray image.

In an example, the processing unit is configured to register the actual 3D ultrasound data to a projection direction of the actual 2D X-ray image.

In an example, generating the graphical representation by the processing unit is provided by transforming the distance value into the graphical representation.

In another example, a device for providing spatial information of an interventional device in an actual 2D X-ray image is provided. The device comprises an input unit and a processing unit. The input unit is configured to provide an actual 2D X-ray image of a region of interest related to a portion of a patient's body. A target location lies within the region of interest. The input unit is further configured to provide actual 3D ultrasound data of at least a part of the region of interest and at least a portion of an interventional device. The processing unit is configured to register the actual 3D ultrasound data to the actual 2D X-ray image. The processing unit is also configured to identify the target location in the 3D ultrasound data, and to determine, in case the interventional device is moved such that portion of the interventional device is arranged within the region of interest, based on the actual 3D ultrasound data, a spatial relation between the portion of the interventional device and the target location. The spatial relation is referred to as a depth value or as a distance value in z-direction of the 2D X-ray image. The processing unit is configured to retrieve a distance value in a projection direction of the actual 2D X-ray image from the spatial relation, and to generate a graphical representation based on the distance value, the graphical representation to be displayed with the actual 2D X-ray image.

In an example, the processing unit is configured to determine the spatial relation between the portion of the interventional device and the target location in case the interventional device is moved such that portion of the interventional device is arranged within the region of interest. The spatial relation is a depth value or a distance value in z-direction of the 2D X-ray image. The processing unit is configured to generate a graphical representation based on the distance value.

According to an example, the distance value in the projection direction of the actual 2D X-ray image is a distance value in z-direction of the 2D X-ray image.

The z-direction is perpendicular to the 2D X-ray image plane.

According to an example, the processing unit is configured to generate image data comprising the actual 2D X-ray image and the graphical representation within the actual 2D X-ray image.

According to an example, the graphical representation is a coded graphical representation. In an option, the coded information, such as a color code or pattern code is indicative for a certain relative distance, like too close towards or too far away from the user, or, in viewing direction of the X-ray projection, behind or in front of the plane in which the target is arranged. In another example, the coded information is indicative for a certain absolute distance, like 10 mm behind or 7.5 mm in front of the target plane.

In an example, the coded graphical representation comprises the coded information in form of the color code or pattern code.

According to an example, a display unit is provided that is configured to display the image data comprising the actual 2D X-ray image and the graphical representation within the 2D X-ray image.

According to an example, the processing unit is configured to identify and to visually indicate the target location in the actual 2D X-ray image.

According to an example, the target location is provided with a marker visible in the actual 3D ultrasound data.

Hence, the device can also be referred to as a device for displaying spatial information of an interventional device in a actual 2D X-ray image.

In an example, the 2D X-ray image is shown on a display unit.

In an example, the display unit is a screen.

In an example, the graphical representation is displayed at the target location.

In an example, a predetermined point of the interventional device can be the tip of the interventional device.

In an example, the interventional device is captured both in 2D X-ray and in 3D ultrasound.

In an example, the interventional device can be a transcatheter device.

In an example, the X-ray and interventional ultrasound coordinate systems are registered by a system.

For example, such registration is provided by a system called EchoNavigator® by Philips.

In an example, the target location can be a main vessel of a heart, into which, for example, a stent is going to be placed with the support of the interventional device.

As a result, a visualization technique is provided that brings useful 3D information to the 2D X-ray image without cluttering up the X-ray image. As X-ray image projections do not show “depth” dimension, this dimension in projection direction is now brought back to the X-ray 2D projection image, i.e. the 2D image is enriched or augmented with relative depth information. That means, the user, e.g. an interventionist, may rely on the 2D X-ray image for navigating the interventional device to the target location. By also retrieving a distance value in the projection direction of the interventional device to the target (i.e. target location, target position or target point), or so-called relative depth, from the spatial relation, and transforming the distance value into a graphical representation, a actual 2D X-ray image is generated showing the movement of the interventional device on the 2D X-ray image with additional visual information on the relative depth information without cluttering up the 2D X-ray image. By this, the user can concentrate on the relevant information and is not distracted. Since the additional information is provided by ultrasound, no further X-ray radiation is required.

In a still further example, a device is provided for providing spatial information of an interventional device in a actual 2D X-ray image. The device comprises an input unit and a processing unit. The input unit is adapted to handle several data from different data sources. It is also configured to handle different data types. The input unit is configured to provide an actual 2D X-ray image of a region of interest. The region of interest is related to a portion of the patient's body in which a user is navigating an interventional device. A predetermined point of the interventional device is to be navigated to a predetermined target location within the region of interest. The input unit is further configured to provide actual 3D ultrasound data of at least a part of the region of interest comprising the predetermined point of the interventional device. The processing unit is configured to register the actual 3D ultrasound data to a projection direction of the actual 2D X-ray image. The processing unit is configured to identify the predetermined target location in the 3D ultrasound data, and to determine, based on the actual 3D ultrasound data, a spatial relation between an actual position of the predetermined point of the interventional device and the predetermined target location at least in the projection direction of the actual 2D X-ray image. In an example, the processing unit identifies the predetermined target location by processing a geometrical command from the user, i.e. surgeon before starting the operation. For example, the user transmits the geometrical information of the patient's body via the input unit to the processing unit of the device. In an example, the processing unit determines the distance value between the target location and the portion of the interventional device once the portion of the interventional device is in the region where it can captured by a 3D ultrasound device. In another example, the processing unit determines the distance value between the interventional device and the target location once both lie within the region of interest. The processing unit is configured to retrieve a distance value in the projection direction of the actual 2D X-ray image from the spatial relation, and transforming the distance value into a coded graphical representation. The processing unit is configured to identify and visually indicate the target location in the actual 2D X-ray image. The processing unit is configured to provide image data for displaying the actual 2D X-ray image and the coded graphical representation.

According to a second aspect, an X-ray imaging system is provided for providing spatial information of an interventional device in a live 2D X-ray image. The system comprises an X-ray imaging device, an ultrasound imaging device and a device for providing spatial information of an interventional device in a live 2D X-ray image according to one of the examples above. The X-ray imaging device provides the actual 2D X-ray image, i.e. the actual 2D X-ray image of the region of interest related to the portion of a patient's body, wherein the target location lies within the region of interest, and wherein at least the portion of the interventional device is arranged in region of interest. The ultrasound imaging device provides actual 3D ultrasound data, i.e. the actual 3D ultrasound data of the part of the region of interest and the portion of the interventional device, wherein the target location lies within the region of interest.

In an example, the region of interest is related to a portion of the patient's body in which a user is navigating an interventional device. A predetermined point of the interventional device is to be navigated to a predetermined target location within the region of interest.

The X-ray imaging system can also be referred to as an X-ray imaging system for displaying spatial information of an interventional device in a live 2D X-ray image.

The term “provide” can also be referred to as capture the X-ray image.

In example, the X-ray imaging device provides the actual 2D X-ray image of a region of interest related to a portion of the patient's body in which a user is navigating an interventional device. A predetermined point of the interventional device is to be navigated to a predetermined target location within the region of interest.

According to an example, the ultrasound imaging device is insertable in a patient's body.

In a still further example, a system is provided for providing spatial information of an interventional device in a live 2D X-ray image. The system comprises an X-ray imaging device, an ultrasound imaging device and a device for providing spatial information of an interventional device in a live 2D X-ray image according to the examples above. The X-ray imaging device provides a actual 2D X-ray image of a region of interest. The region of interest is related to a portion of the patient's body in which a user is navigating an interventional device. A predetermined point of the interventional device is to be navigated to a predetermined target location within the region of interest. The ultrasound imaging device provides actual 3D ultrasound data of at least a part of the region of interest comprising the predetermined point of the interventional device.

According to third aspect, a method for providing spatial information of an interventional device in a live 2D X-ray image is provided. The method comprises the following steps:

a) In a first step, an actual 2D X-ray image of a region of interest is provided, being related to a portion of the patient's body, wherein a target location lies within the region of interest. At least a portion of an interventional device is arranged in the region of interest.
b) In a second step, actual 3D ultrasound data of at least a part of the region of interest and at least a portion of an interventional device is provided.
c1) In a third step, the actual 3D ultrasound data is registered to the actual 2D X-ray image.
c2) In a fourth step, the target location in the 3D ultrasound data is identified and the spatial relation between the portion of the interventional device and the target location is determined based on the actual 3D ultrasound data. Further, a distance value in a projection direction of the actual 2D X-ray image is retrieved from the spatial relation.
c3) In fifth step, based on the distance value, a graphical representation indicative of the spatial relation to be displayed with the actual 2D X-ray image is generated.

In an example, a further (sixth) step is provided, also referred to as step d), in which the target location in the actual 2D X-ray image is identified and visually indicated.

In another (seventh) step, also referred to as step e), image data for displaying the actual 2D X-ray image and the graphical representation is provided.

In an example, the region of interest is related to a portion of a patient's body in which a user is navigating an interventional device. For example, a predetermined point of the interventional device is to be navigated to a predetermined target location within the region of interest.

In an example, the region of interest is comprising the predetermined point of the interventional device.

In an example, in step c1), it is provided a step of registering the actual 3D ultrasound data to a projection direction of the actual 2D X-ray image.

In an example, in step c2), it is provided a step of identifying the predetermined target location in the 3D ultrasound data; and determining, based on the actual 3D ultrasound data, a spatial relation between an actual position of the predetermined point of the interventional device and the predetermined target location at least in the projection direction of the actual 2D X-ray image.

The graphical representation may be a coded graphical representation.

In an example, in step e) or following step e), the image data comprising the actual 2D X-ray image and the graphical representation is displayed.

In an example, steps a) to e) are provided in a synchronous continuous manner, such that the user has instant actual data of the area of interest in the patient's body.

In an example, the distance value is determined in 3D ultrasound, e.g. the distance between the point of the interventional device and the target location. For example, the distance value between the point of the interventional device and the target location in the projection direction is computed by subtracting the distance of the point of the interventional device to the X-ray source from the distance of the target location to the X-ray source or vice versa.

In an example, the relative depth is computed as the distance between the target location and a projection of the target point of the interventional device onto a line that goes through both the target and the X-ray imaging device. The projection may, for example, be the standard Euclidean projection of the point of the interventional device.

In another example, the closest point to the line going through the point of the interventional device is provided and aligned with the interventional device, assuming an estimation of the orientation of the interventional device is available additionally to the position of the point of the interventional device.

Hence, the method can also be referred to as a method for displaying spatial information of an interventional device in an actual or live 2D X-ray image.

According to an example, the graphical representation is displayed at the target location.

In an example, a first circle is indicating the target location in the 2D X-ray image, and the first circle changes its color according to a color coding.

In an example, the color coding indicates whether the interventional device is too close or too far from the target location.

According to another example, it is provided a step d) that comprises identifying and visually indicating the position of the interventional device in the actual 2D X-ray image. It is further provided a step e) that comprises that the graphical representation is displayed at the position of the interventional device.

In an option, the graphical representation follows a change of the position of the interventional device within the actual 2D X-ray image.

In a still further example, a method for providing spatial information of an interventional device in a live 2D X-ray image is provided. The method comprises the following steps. In a first step, also referred to as step a), an actual 2D X-ray image of a region of interest is provided, wherein the region of interest is related to a portion of the patient's body in which a user is navigating an interventional device. A predetermined point of the interventional device is to be navigated to the predetermined target location within the region of interest. In a second step, also referred to as step b), actual 3D ultrasound data of at least a part of the region of interest comprising the predetermined point of the interventional device is provided. In a third step, also referred to as step c1), the actual 3D ultrasound data is registered to a projection direction of the actual 2D X-ray image. In a fourth step, also referred to as step c2), the predetermined target location in the 3D ultrasound data is identified and the spatial relation between an actual position of the predetermined point of the interventional device and the predetermined target location at least in a projection direction of the actual 2D X-ray image is determined based on the actual 3D ultrasound data. In a fifth step, also referred to as step c3), a distance value in the projection direction from the spatial relation is retrieved, and the distance value is transformed into a coded graphical representation. In a sixth step, also referred to as step d) the target location in the actual 2D X-ray image is identified and visually indicated. In a seventh step, also referred to as step e), image data for displaying the actual 2D X-ray image and the coded graphical representation is provided.

According to an aspect, it is provided to visualize the distance of an interventional device to a target perpendicular to the actual 2D X-ray image plane by a graphical representation. By this, a user is provided with information about the positioning of the interventional device in the direction perpendicular to the 2D image, which can also be called the depth direction. For example, when navigating an interventional device to a target point, on the actual 2D X-ray image, the user is provided with the position of the interventional device to the target in the 2D plane. The orientation of the interventional device in the depth direction is visualized by a graphical representation, like a circle with a color coding indicating whether the interventional device is far or close to the target point. In addition to the X-ray images, interventional ultrasound thus provides useful additional information relating to the anatomy and the interventional tool in 3D. As a result, a fusion of information relating from X-ray and from interventional ultrasound is achieved. The different geometric natures of these modalities, the former being projective and the latter volumetric, is combined to present the combined or fused information to the user. The 3D information from ultrasound imaging is merged with the projective X-ray image to provide a 2D image (the X-ray projection image) that also comprises 3D information (from the ultrasound).

These and other aspects of the present invention will become apparent from and be elucidated with reference to the embodiments described hereinafter.

