[ 0027 ] fig1 illustrates an image - guided , robotic - assisted surgery system , which may be used to implement embodiments of the present invention . the system includes a surgical or medical instrument 12 having an elongate axis 14 and a tip 16 . in one embodiment , the instrument may be a viewing instrument , such as an endoscope or surgical microscope , equipped with a lens for viewing an internal target site 18 and any intervening tissue 19 of a patient 20 . in another embodiment , the instrument is preferably a probe , such as an ultrasound probe for tracking a moving target inside the patient &# 39 ; s body . the instrument may also include a pointer or a tool , such as a drill . in accordance with embodiments of the invention , instrument 12 is releasably attached to the distal - end of an end arm segment 22 of a processor - controlled , motor - driven , multi - arm assembly 24 . the assembly is preferably a robotic - arm assembly with one or more fine control motors for precisely controlling movement of the individual arm segments , which are interconnected by universal joints 26 or the like . typically , there will be one less universal joint than arm segments . the first arm segment of the robotic - arm assembly is attached to a base 28 . the robotic - arm assembly may be an articulated arm , a haptic device , or a cobotic device . descriptions of cobotic devices may be found , for example , in u . s . pat . no . 5 , 952 , 796 . before the tracking procedures of the present invention are implemented , the patient &# 39 ; s target site is registered to images of the site . this may be accomplished in a variety of ways . in one embodiment , a plurality of fiducial markers 30 placed on the patient near the target site are used to register corresponding points on preoperative or intraoperative 2 - d image scans of patient target site 18 . corresponding points are those points that represent the same anatomical features in the two spaces . in general , there are two types of registration image - to - image and image - to - physical . the algorithms employed to accomplish registration are mathematically and algorithmically identical in each case . they use as input the 3 - d positions of three or more fiducials in both spaces , and they output the point - for - point mapping from one space to another . the mapping addresses the physical differences in position of the two spaces , which consists of a shift , a rotation , a scale or a combination thereof . the correct mapping , or registration , is the particular rotation , shift or scale that will map all the localized fiducial positions in one 3 - d space , for example , the physical space around the patient in the operating room , to the corresponding localized positions in the second space , for example , a ct image . if these fiducial positions are properly mapped then , unless there is distortion in the images , all non - fiducial points in the first space will be mapped to corresponding points in the second space as well . these non - fiducial points are the anatomical points of interest to the surgeon . because of inevitable small errors in the localization of the fiducial points , it is rarely possible to find a rotation , a shift or a scale that will map all fiducial points exactly from one space to the other . therefore , an algorithm is used that finds the rotation , shift or scale that will produce the smallest fiducial mapping error ( in the standard least - squares sense ). this mapping error provides a measure of the success of the registration . it is computed by first calculating , for each fiducial , the distance between its localized position in the second space and the localized position in the first space as mapped into the second space . the mapping error is then computed by calculating the square root of the average of the squares of these distances . in one embodiment , a computer system is used to render and display the 2 - d preoperative images and render 3 - d volumetric perspective images of target site 18 on a display device . registration is then accomplished by successively pointing or touching the tip of the instrument to each of the fiducial markers on the patient , moving the computer cursor onto the corresponding image fiducial , and activating an appropriate input device ( e . g ., clicking a mouse or foot pedal ) to map the physical fiducial to the image fiducial . this may be done before or after the instrument is attached to the robot . if done before instrument attachment , instrument 12 will have associated with it a mechanism for tracking the instrument . for example , the instrument can be equipped with a plurality of tracking elements 32 on its shaft 14 which emit signals to sensors 34 positioned in view of the instrument . both the instrument and the sensors will be in communication with a tracking controller , which is in communication with the computer system that processes the signals received by sensors 34 in carrying out the registration process . alternatively , registration may be done with the instrument attached to the robot , since the robot is in two - way communication with the tracking controller . as previously noted , the registration procedure described above is merely one way of carrying out the registration process . other ways known in the art may also be employed . during the surgical procedure , with the instrument attached to the robot , the instrument &# 39 ; s position and orientation is known with respect to the robot &# 39 ; s coordinate system . thus , by processing the signals received from the robot through the tracking controller , the computer system is able to track the movement of instrument 12 . the instrument may also be tracked using tracking elements 32 . the tracking controller may be a separate element or it may be physically integrated with the computer system and may even be embodied in an option card which is inserted into an available card slot in the computer . various aspects of the image - guided , robotic - assisted surgery procedure , including tracking , control of the robotic - arm assembly to enforce a desired orientation of the instrument , and image rendering , may be implemented by a program of instructions ( e . g ., software ) based on initial user input which may be supplied by various input devices such as a keyboard and mouse . software implementing one or more of the various aspects of the present invention may be written to run with existing software used for image - guided surgery . the software for such tasks may be fetched by a processor , such as a central processing unit ( cpu ), from random - access memory ( ram ) for execution . other processors may also be used in conjunction with the cpu such as a graphics chip for rendering images . the software may be stored in read - only memory ( rom ) on the computer system and transferred to ram when in use . alternatively , the software may be transferred to ram , or transferred directly to the appropriate processor for execution , from rom , or through a storage medium such as a disk drive , or through a communications device such as a modem or network interface . more broadly , the software may be conveyed by any medium that is readable by the processor . such media may include , for example , various magnetic media such as disks or tapes , various optical media such as compact disks , as well as various communication paths throughout the electromagnetic spectrum including infrared signals , signals transmitted through a network or the internet , and carrier waves encoded to transmit the software . as an alternative to software implementation , the above - described aspects of the invention may be implemented with functionally equivalent hardware using discrete components , application specific integrated circuits ( asics ), digital signal processing circuits , or the like . such hardware may be physically integrated with the computer processor ( s ) or may be a separate device which may be embodied on a computer card that can be inserted into an available card slot in the computer . thus , the above - mentioned aspects of the invention can be implemented using software , hardware , or combination thereof . the disclosure provides the functional information one skilled in the art would require to implement a system to perform the functions required , with software , functionally equivalent hardware , or a combination thereof . [ 0043 ] fig2 is a flow chart illustrating the process of setting up the robotic tracking in accordance with embodiments of the invention . first , the preoperative or intraoperative scan data representing internal scans of the patient target site are acquired and used to construct various 2 - d images taken in different planes and a 3 - d image of the patient target site . these images are displayed on the display device for viewing by the user . the user then assigns an “ image ” target point 40 on the 2 - d images by , for example , pointing the computer cursor at the desired location on the images and inputting information to the computer ( e . g ., by clicking a mouse or foot pedal ) to establish that point as the image target point . the computer establishes a correspondence between assigned target point 40 and a target point 42 in the patient &# 39 ; s body by , for example , using point - to - point mapping as is done in the registration procedure . point - to - point mapping essentially involves determining a transformation matrix that maps the coordinates of point 42 to another set of coordinates representing point 40 . the computer stores the target point coordinate data in a storage media , such ram , rom or disk . next , the robot is tracked , as the predetermined task is carried out by the robot . in the first embodiment , the task of the robot is to make the necessary adjustments to keep the viewing instrument directed toward the target point , as the surgeon moves the instrument in space to determine the optimal point of entry to the target site within the patient &# 39 ; s body . for example , as the surgeon grasps the end segment 22 and applies a force ( f ) to it to move the tip of the instrument from point x 1 to point x 2 , as shown in fig3 the computer determines the appropriate correction to be applied , and the tracking controller sends signals to the robot to activate its internal motors to move one or more of the arm segments to reorient the axis of the instrument toward the direction of target point 42 . this correction , while not instantaneous , is made as the surgeon moves the end arm segment to quasi - continuously maintain colinearity between the axis of the instrument and target point 42 . the instrument is a medical instrument , such as a viewing instrument ( e . g ., an endoscope ) adapted to generate image signals indicative of the view along the axis of the instrument and to transmit such signals to the tracking controller which , in turn , sends the signals to the computer system which processes the signals and renders on the display an image of the patient &# 39 ; s target site and any intervening tissue , as viewed along the axis of the instrument . an exemplary endoscope is illustrated in fig7 a . the endoscope 112 has an elongate axis 114 and a base 115 that fits into an appropriately sized bore in the distal end of end arm segment 22 . the base contains circuitry to transmit images captured by the endoscope through its lens 117 . a fiber optic cable 121 and a video cable 123 interface with the endoscope through an adapter 125 to transmit signals to the tracking controller and on to the computer system , as is known in the art . [ 0047 ] fig4 is a flow chart showing the interactive robot correction process according to the first embodiment of the invention . with the instrument in a present state with its axis aligned with the target point , a user applies a force either to the instrument itself or to the end arm segment of the robot to move the tip of the instrument from one point to another . the computer determines if the applied force has moved the axis of the instrument off - trajectory with respect to the target point and also determines the appropriate correction required by analyzing the signals received from the robot indicative of the position and orientation of the instrument and comparing this data with the target point coordinate data stored in memory . the tracking controller , who is in continuous two - way communication with the computer , then sends signals to the robot to activate its motors to carry out the correction . in accordance with a second embodiment , the medical instrument is a surgical tool that has a pressure sensor / transducer or the like in the tip of the tool . the tool is preferably an ultrasonic probe , for example , as shown in fig7 b . the ultrasound probe has an elongate portion 224 , one end of which fits in a bore in the distal end of end arm segment 22 . the other end of the probe terminates in a head 227 that has pressure or force contact sensors 250 positioned therein . the sensors are positioned so that the contact surface of the transdu are approximately flush with the contact surface of the probe head . as schematically shown in fig7 b , the sensors are in communication with the processor circuitry that controls robotic assembly 24 to provide a feedback signal indicative of the pressure or contact between the probe and a tissue surface . the probe further includes an image array 260 that tracks a moving target in its field of view . appropriate communication paths may be provided so that the images obtained by the image array may be processed by the computer system and displayed . this second embodiment is similar to the first embodiment in that the probe &# 39 ; s orientation is enforced along the axis of the probe toward the target point . here , however , the surgeon does not move the probe ; instead , the robot applies the only driving force on the probe to track a moving target , such as the tip of a biopsy needle , inside the body , while the tip of the probe is maintained at a substantially constant pressure against a tissue surface . the tip of the probe is fixed , and the robot is actuated to move the proximal end of the end arm segment to maintain colinearity between the axis of the probe and the target point , as the target moves . simultaneously , the pressure sensor ( s ) in the probe tip provide feedback signals to the robot in order to maintain the substantially constant pressure between the probe and tissue surface . during the entire targeting and scanning procedure , the position and the pressure of the probe tip remains constant , as illustrated in fig5 . as is the case with the correction in the previous embodiment , this correction , while not instantaneous , is made on a real - time basis . the target can be tracked via a 3 - d localizer or through image processing , i . e ., viewing the target on an image . [ 0051 ] fig6 is a flow chart illustrating the tracking process according to the second embodiment of the invention . with the probe in an initial state with its axis aligned with the target point and its tip held against a tissue surface at a constant , predetermined pressure , the target point moves within the patient &# 39 ; s body . as this occurs , the computer updates the coordinates of the target point , determines if the axis of the probe is off - trajectory with respect to the “ new ” target point coordinates , and determines the appropriate correction required by comparing the “ present ” position and orientation of the instrument data with the updated target point coordinate data . the tracking controller , who is in continuous communication with the computer , then sends signals to the robot to carry out the correction . while this correction is being carried out , the pressure transducer in the probe tip is also sending feedback signals to the robot to maintain the predetermined pressure between the tissue surface and the probe tip . this embodiment has various applications . for example , the ultrasonic probe may be used to track a point ( e . g ., the tip ) of a moving biopsy , as it is approaching a targeted lesion inside the body . while embodiments of the invention have been described , it will be apparent to those skilled in the art in light of the foregoing description that many further alternatives , modifications and variations are possible . the invention described herein is intended to embrace all such alternatives , modifications and variations as may fall within the spirit and scope of the appended claims .
an apparatus and method for adjusting the orientation of a surgical viewing instrument , which may be used to view a patient target site and any intervening tissue from outside the body , as the position of the instrument is changed by a user . the instrument is attached to a robotic arm assembly and is movable by both the user and the robot . as the user moves the instrument to a different position , the robot automatically corrects the orientation of the instrument to maintain a viewing trajectory defined by the axis of the instrument and a target coordinate in the patient target site . in another aspect there is an apparatus and method for using a surgical robot and attached ultrasound probe to track a moving target in a patient &# 39 ; s body . the ultrasound probe has a pressure sensor in its tip , which is maintained in contact with a tissue surface at a specific location at a constant pressure . subject to this constraint , the robot is directed to adjust the orientation of the probe , as the target point moves , to maintain the axis of the probe in line with the target point .