TECHNICAL FIELD

Aspects of this disclosure are generally related to human-machine interfaces, and more particularly to cursors.

BACKGROUND

The typical arrow-shaped cursor presented by a computer operating system is zero-dimensional. A zero-dimensional cursor designates the location of a single point in a space such as a two-dimensional window presented on a monitor. Mouse buttons can be used in combination with movement of the cursor to select objects in the two-dimensional space, but at any given instant of time a zero-dimensional cursor position designates only a single point in space.

The current standard for diagnostic radiologists reviewing computed tomography (CT) or magnetic resonance imaging (MRI) studies is a slice-by-slice method. A conventional keyboard, monitor, and mouse with a zero-dimensional cursor are used for manipulating the images. The use of mouse buttons and cursor movement for manipulating the images can become burdensome. For example, many images are included in radiology studies that are performed for the follow up of cancer to determine the response to treatment. The ability to recognize and analyze differences between images can be important. As an example, the recent Investigation of Serial Studies to Predict Your Therapeutic Response with Imaging and Molecular Analysis (I-SPY) trial tracked the changes in the tumor over multiple magnetic resonance imaging (MRI) scans during the administration of neoadjuvant chemotherapy (NACT). It has been noted that the phenotypic appearance (e.g., shape, margins) of a tumor correlated with the pathologic response to NACT. A more efficient and accurate interface for manipulating and presenting medical images would therefore have utility.

Known techniques for 3D viewing of medical images are described in U.S. Pat. No. 9,349,183, Method and Apparatus for Three Dimensional Viewing of Images, issued to Douglas, U.S. Pat. No. 8,384,771, Method and Apparatus for Three Dimensional Viewing of Images, issued to Douglas, Douglas, D. B., Petricoin, E. F., Liotta L., Wilson, E. D3D augmented reality imaging system: proof of concept in mammography.Med Devices(Auckl), 2016; 9:277-83, Douglas, D. B., Boone, J. M., Petricoin, E., Liotta, L., Wilson, E. Augmented Reality Imaging System: 3D Viewing of a Breast Cancer.J Nat Sci.2016; 2(9), and Douglas, D. B., Wilke, C. A., Gibson, J. D., Boone, J. M., Wintermark, M. Augmented Reality: Advances in Diagnostic Imaging.Multimodal Technologies and Interaction,2017; 1(4):29. In D3D imaging, the radiologist wears an augmented reality (AR), mixed reality (MR) or virtual reality (VR) headset and uses a joystick or gaming controller. Advantages include improved depth perception and human machine interface. Still, there are several challenges faced with this approach. First, an area of interest (e.g. tumor) may be in close proximity to structures that are similar in composition/density. Isolating the area of interest for better examination may be difficult. Second, many soft tissues in the body are mobile and deformable, so it can be difficult to achieve the best orientation to properly compare the tumor at multiple time points. Efficiently aligning the orientation to do so may be difficult. Third, certain portions of a tumor can respond to treatment and decrease in size while other portions of a tumor demonstrate increases in size. The pattern of tumor shrinkage has important prognostic implications. Furthermore, composition and complex morphologic features including spiculations (spikes extending from the surface), irregular margins and enhancement also have important implications. Consequently, there is a need for a system that facilitates recognition of the subtle, yet important changes in size, shape and margins. Fourth, a patient with metastatic cancer has several areas of interest in different areas of the body. It is difficult and time consuming to find each of the areas of interest at every time point to determine interval change. Consequently, there is a need for a system that enables the observer to do this efficiently.

SUMMARY

All examples, aspects and features mentioned in this document can be combined in any technically possible way.

In accordance with an aspect of the invention a method comprises: generating a three-dimensional cursor that has a non-zero volume; responsive to a first input, moving the three-dimensional cursor within a three-dimensional image; responsive to a second input, selecting a volume of the three-dimensional image designated by the three-dimensional cursor; and responsive to a third input, presenting a modified version of the selected volume of the three-dimensional image. In some implementations presenting the modified version of the selected volume of the three-dimensional image comprises removing an un-selected volume of the three-dimensional image from view. In some implementations presenting the modified version of the selected volume of the three-dimensional image comprises changing transparency of presented tissues within the selected volume. In some implementations presenting the modified version of the selected volume of the three-dimensional image comprises filtering a selected tissue to remove the selected tissue from view. In some implementations presenting the three-dimensional cursor with measurement markings on at least one edge, surface or side. In some implementations presenting the modified version of the selected volume of the three-dimensional image comprises presenting inputted location indicators. In some implementations presenting the modified version of the selected volume of the three-dimensional image comprises presenting inputted annotations. Some implementations comprise changing a size dimension of the three-dimensional cursor responsive to a fourth input. Some implementations comprise changing a geometric shape of the three-dimensional cursor responsive to a fifth input. Some implementations comprise automatically generating a statistical representation of the selected volume of the three-dimensional image. In some implementations presenting the modified version of the selected volume of the three-dimensional image comprises presenting at least one tissue type with false color. In some implementations presenting the modified version of the selected volume of the three-dimensional image comprises presenting volumetric changes over time with false color. Some implementations comprise presenting multiple computed tomography images associated with the selected volume using reference lines. Some implementations comprise presenting multiple axial computed tomography images associated with the selected volume using reference lines. Some implementations comprise presenting a maximum intensity projection (MIP) image of a positron emission tomography (PET) scan with the three-dimensional cursor overlaid thereon to indicate orientation and location of the selected volume. Some implementations comprise presenting a radiology report enhanced with information obtained using the three-dimensional cursor. Some implementations comprise automatically calculating and presenting a quantitative analysis and a qualitative analysis associated with multiple time points. Some implementations comprise presenting the modified version of the selected volume of the three-dimensional image comprises presenting inputted registration markers. Some implementations comprise automatically calculating volumetric change based on the registration markers. Some implementations comprise automatically re-orienting the selected volume of the three-dimensional image based on the registration markers. Some implementations comprise using multiple volumes selected with the three-dimensional cursor to designate a pre-operative planning pathway for guiding surgical intervention. Some implementations comprise presenting the selected volume with an augmented reality, virtual reality or mixed reality headset.

In accordance with an aspect of the invention an apparatus comprises: a computing device; and a human-machine interface comprising a three-dimensional cursor that has a non-zero volume; the human-machine interface moving the three-dimensional cursor within a three-dimensional image responsive to a first input; the human-machine interface selecting a volume of the three-dimensional image designated by the three-dimensional cursor responsive to a second input; and the human-machine interface presenting a modified version of the selected volume of the three-dimensional image responsive to a third input. In some implementations, the human-machine interface removes an un-selected volume of the three-dimensional image from view. In some implementations, the human-machine interface changes transparency of presented tissues within the selected volume. In some implementations, the human-machine interface filters a selected tissue to remove the selected tissue from view. In some implementations, the human-machine interface presents the three-dimensional cursor with measurement markings on at least one edge, surface or side. In some implementations, the human-machine interface receives and implements inputted location indicators. In some implementations, the human-machine interface receives and implements inputted annotations. In some implementations, the human-machine interface changes a size dimension of the three-dimensional cursor responsive to a fourth input. In some implementations, the human-machine interface changes a geometric shape of the three-dimensional cursor responsive to a fifth input. In some implementations, the human-machine interface automatically generates and presents a statistical representation of the selected volume of the three-dimensional image. In some implementations, the human-machine interface presents at least one tissue type with false color. In some implementations, the human-machine interface presents volumetric changes over time with false color. In some implementations, the human-machine interface presents multiple computed tomography images associated with the selected volume using reference lines. In some implementations, the human-machine interface presents multiple axial computed tomography images associated with the selected volume using reference lines. In some implementations, the human-machine interface presents a maximum intensity projection (MIP) image of a positron emission tomography (PET) scan with the three-dimensional cursor overlaid thereon to indicate orientation and location of the selected volume. In some implementations, the human-machine interface presents a radiology report enhanced with information obtained using the three-dimensional cursor. In some implementations, the human-machine interface automatically calculates and presents a quantitative analysis and a qualitative analysis associated with multiple time points. In some implementations, the human-machine interface presents inputted registration markers. In some implementations, the human-machine interface automatically calculates volumetric change after appropriate registration using the registration markers. In some implementations, the human-machine interface automatically re-orients the selected volume of the three-dimensional image based on the registration markers. In some implementations, the human-machine interface presents multiple volumes selected with the three-dimensional cursor to designate a pre-operative planning pathway for guiding surgical intervention. In some implementations, the human-machine interface presents the selected volume with an augmented reality, virtual reality or mixed reality headset.

