BACKGROUND

In general, a simulation provides representations of certain key characteristics or behaviors of a selected physical or abstract system. Simulations can be used to show the effects of particular courses of action. A physical simulation is a simulation in which physical objects are substituted for a real thing or entity. Physical simulations are often used in interactive simulations involving a human operator for educational and/or training purposes. For example, mannequin patient simulators are used in the healthcare field, flight simulators and driving simulators are used in various industries, and tank simulators may be used in military training.

Physical simulations or objects provide a real tactile and haptic feedback for a human operator and a 3-dimensional (3D) interaction perspective suited for learning psycho-motor and spatial skills.

In the health care industry, as an example, medical simulators are being developed to teach therapeutic and diagnostic procedures, medical concepts, and decision making skills. Many medical simulators involve a computer or processor connected to a physical representation of a patient, also referred to as a mannequin patient simulator (MPS). These MPSs have been widely adopted and consist of an instrumented mannequin that can sense certain interventions and, via mathematical models of physiology and pharmacology, the mannequin reacts appropriately in real time. For example, upon sensing an intervention such as administration of a drug, the mannequin can react by producing an increased palpable pulse at the radial and carotid arteries and displaying an increased heart rate on a physiological monitor. In certain cases, real medical instruments and devices can be used with the life-size MPSs and proper technique and mechanics can be learned.

Physical simulations or objects are limited by the viewpoint of the user. In particular, physical objects such as anesthesia machines (in a medical simulation) and car engines (in a vehicle simulation) and physical simulators such as MPSs (in a medical simulation) remain a black-box to learners in the sense that the internal structure, functions and processes that connect the input (cause) to the output (effect) are not made explicit. Unlike a user's point of reference in an aircraft simulator where the user is inside looking out, the user's point of reference in, for example, a mannequin patient simulator is from the outside looking in any direction at any object, but not from within the object.

In addition, many visual cues such as a patient's skin turning cyanotic (blue) from lack of oxygen are difficult to simulate. These effects are often represented by creative substitutes such as blue make-up and oatmeal vomit. However, in addition to making a mess, physically simulated blood gushing from a simulated wound or vomit can potentially cause short-circuits because of the electronics in a MPS.

Furthermore, it can be difficult to create a physical simulator having internal features that can be repeatedly interacted with. For example, physical simulators or mannequins for insertion of needles or lines into veins and/or arteries often involve plastic tubing carrying colored liquids. These plastic tubes have a limited life span due to puncturing and leaks. Often, disposable components and replacement parts are used in order to carry out multiple sessions on a single physical simulator or mannequin. In addition, mass production of physical disposables is less amenable to implementation of anatomical variability than a virtual model. Moreover, these disposable components and replacement parts can be expensive, thereby limiting the number of times a person can use the simulator in a cost-effective manner. In addition, the short lifetime of these disposable components and replacement parts leads to increased volume disposed in a landfill.

Virtual simulations have also been used for education and training. Typically, the simulation model is instantiated via a display such as a computer, PDA or cell phone screen; or a stereoscopic, 3D, holographic or panoramic display. An intermediary device, often a mouse, joystick, or Wii™, is needed to interact with the simulation.

Virtual abstract simulations, such as transparent reality simulations of anesthesia machines and medical equipment or drug dissemination during spinal anesthesia, emphasize internal structure, functions and processes of a simulated system. Gases, fluids and substances that are usually invisible or hidden can be made visible or even color-coded and their flow and propagation can be visualized within the system. However, in a virtual simulation without the use of haptic gloves, the simulator cannot be directly touched like a physical simulation. In the virtual simulations, direct interaction using one's hands or real instruments such as laryngoscopes or a wrench is also difficult to simulate. For example, it can be difficult to simulate a direct interaction such as turning an oxygen flowmeter knob or opening a spare oxygen cylinder in the back of the anesthesia machine.

In addition, important tactile and haptic cues, such as the deliberately fluted texture of an oxygen flowmeter knob in an anesthesia machine or the pressure in a needle/syringe felt when moving through fat and muscle or impinging on bone structures, are missing. Furthermore, the emphasis on internal processes and structure may cause the layout of the resulting virtual simulation to be abstracted and simplified and thus different from the actual physical layout of the real system. This abstract representation, while suited for assisting learning by simplification and visualization, may present challenges when transferring what was learned to the actual physical system.

Accordingly, there continues to be a need for a simulation system capable of in-context integration of virtual representations with a physical simulation or object.

BRIEF SUMMARY

The present invention provides an interactive combined physical and virtual (i.e., mixed) system. According to embodiments of the present invention, part or all of a 3D virtual model is registered invisibly and/or internally to a physical model. This invisible and/or internally registered 3D virtual model may not be visible to a user or operator of the subject interactive system when performing a simulation. The user or operator can interact with the physical model via at least one of different methods, such as palpation, tactile feedback, and a physical tool or implement that may be tracked in a 3D space. Embodiments of the invention can be used to simulate “blind” procedures where an operator is interacting with an object without direct line of sight for what he or she is doing.

In accordance with certain embodiments of the invention, only part or all of the surface or external visible area of the physical model needs to be implemented for purposes of the simulation, while part or all of the internal structure, processes, and functions inside the model may be implemented virtually. For such embodiments, the time and effort to manufacture and precisely locate part or all of the physical representations of the internal structure, processes, and functions in 3D-space is eliminated, as well as any expense and environmental impact associated with physical disposables.

The present invention provides, as an example, mixed simulator systems capable of providing training for blind procedures where direct line of sight is not available. The mixed simulator system of an embodiment of the invention combines a physical object and/or simulator with a virtual representation and/or simulator. Accordingly, it is possible to simultaneously interact with a physical object and a virtual simulation of internal elements of the physical object.

In one embodiment for a medical implementation, the subject interactive mixed simulator system incorporates a patient mannequin, a tool and/or instrument to interact with the patient mannequin, and a virtual model of internal anatomy that can be associated with the patient mannequin and instantiated on a display. The patient mannequin can be a partial mannequin or a full body representing a patient, and can include physical body characteristics, such as anatomical landmarks of a patient. The tool and/or instrument to interact with the patient mannequin can be, for example, an actual or fake medical instrument, a haptic device used in a physical simulation, or a combination thereof. The virtual model can be instantiated on a screen or monitor of a display device and/or registered to its physical counterpart that may be produced from a 3D printer or fast prototyping machine. The display device can be mobile, such as a computer tablet, or stationary, such as a computer or television screen.

According to one medical implementation, a mixed simulator is provided that can be used to train for insertion of a central venous line (CVL). A trainer for central venous access can include a 3D virtual model created using a medical scan of a patient. A library of patient anatomies is provided to reflect anatomical variability for different ages, genders, ethnicities and body types. A physical mannequin and a physical syringe are also provided for CVL training. For the physical mannequin, a rapid prototyping machine, such as a 3D printer, can be used to create a 3D physical bone structure that, when draped with fake skin, enables visual and tactile (palpation) location of landmarks such as the clavicle and sternal notch. Different thicknesses of sponge layers can be placed below the skin to simulate varying amounts of subcutaneous body fat. The fake skin can be selected to provide the correct tactile feedback upon puncture with a real needle and is placed over the replicated bone structure to create the physical mannequin. In one embodiment, certain parts of the body's external surface (e.g., the skin) can be printed (e.g., by 3-D printing) to provide landmarks while other parts of the body's surface may be covered with simulated skin.

The syringe and physical mannequin are both tracked using an augmented reality tool kit and have model counterparts in a virtual environment. Important “wet” anatomic components, such as lungs, arteries, and veins, can be included in the virtual environment. The user's CVL placement technique is mirrored in the virtual environment where the user's techniques can be analyzed, recorded, and scored. During the simulation, when the tip of the physical needle contacts the physical 3D bone structure, haptic feedback is provided to the operator. This haptic feedback can be provided solely from interaction with the physical 3D bone structure or additional haptic feedback can be provided. In addition, when the operator lets the needle tip stray into the virtual 3D lungs, an iatrogenic pneumothorax can be simulated, a capability that is non-existent in current CVL trainers.

Specifically exemplified herein is an interactive mixed reality system for healthcare training and procedures. It will be clear, however, from the descriptions set forth herein that the mixed reality system of the subject invention finds application in a wide variety of healthcare, education, military, and industry settings including, but not limited to, simulation centers, educational institutions, vocational and trade schools, museums, and scientific meetings and trade shows.

