TECHNICAL FIELD

This disclosure relates to obtaining a moving image for an endoscopic procedure.

BACKGROUND

Endoscopes are commonly used in the medical field for providing access to body cavities with decreased invasiveness. Rigid endoscopes include a rigid insertion tube, while flexible endoscopes typically include a flexible insertion tube. The insertion tubes, which extend from a hand piece or control portion, are configured to be inserted into a body cavity. A distal tip of the insertion tube includes an array of peripherals for providing various functions. For example, the distal tip may include lighting for illuminating the body cavity, one or more ports for delivering air or irrigation, suction ports for removing debris from the body cavity, a medical device or tool, and optics for providing a view of the body cavity to an operator of the endoscope.

Traditionally, the optics of the endoscope were provided mechanically, whereby lenses disposed in the distal tip of the insertion tube were in communication with an eyepiece on the hand piece of the endoscope via optical fiber cable or a series of rod lenses. However, modern endoscopes are often provided with electrical-based optics having one or more camera sensors disposed either at the distal tip of the insertion tube or on a proximal end of the insertion tube via a camera head that couples with a traditional optical endoscope. The camera sensors capture and deliver image signals to a camera control unit (CCU), which processes the image signals and outputs a video stream to a digital display.

Rolling shutter, complementary metal oxide semiconductor (CMOS) sensors are commonly employed in endoscopes due to their ability to provide reduced readout times and faster frame rates. The performance of rolling shutter CMOS systems can be affected by radiation pulses encountered during some endoscopic procedures. For example, during lithotripsy a medical device (e.g., a lithotripter) is used to provide pulses of focused, high-intensity shock waves (e.g., pressure waves) and/or electromagnetic radiation (e.g., laser) to break up stones that typically form in the kidney, bladder, ureters, or gallbladder. The pulses produced by the lithotripter may affect the imaging sensor in the imaging endoscope. For example, when electromagnetic radiation generated by the lithotripter is back-scattered to the imaging sensor, the radiation can temporarily overexpose many of the photo-sensitive elements (e.g., pixels) in the imaging sensor. Although the pulse is typically short in duration, any rows of a frame that are processed during the period of overexposure will consequently appear too bright on the display, a phenomenon known as banding. Accordingly, one or more frames of the video stream may have banding corresponding to rows of the frame that were affected by the radiation pulse.

Banding in the frame of a video stream can be adjusted by using known temporal filters, such as medial temporal filters or median temporal filters. However, temporal filters globally affect the frame, and often result in undesirable artifacts in the video stream, such as blurring or ghosting. Accordingly, there exists a need for modifying an endoscopic image stream to locally correct overexposure caused by radiation pulses.

SUMMARY

A system and method includes operations and steps for obtaining a moving endoscopic image. An optical device stream is received from an optical device by data processing hardware. The data processing hardware identifies image frames of the image stream, each including a plurality of rows of pixels. The data processing hardware determines a row exposure value for each of the rows of pixels in each frame, and identifies a defective image frame having at least one overexposed row and a reference frame having a replacement row corresponding to the overexposed row. The data processing hardware modifies the defective image frame by replacing the overexposed row with the corresponding replacement row of the reference frame.

One aspect of the disclosure provides a method. The method includes receiving, at data processing hardware, an image stream from an optical device. The data processing hardware then identifies image frames of the optical device stream, each of the image frames having an array of pixels including a plurality of rows. The data processing hardware determines a row exposure value for each of the rows in each of the image frames in the image stream, and then identifies a defective image frame including an overexposed row where differences among the row exposure values of neighboring rows of an image frame exceed a threshold value. In one embodiment of the method, the data processing hardware further identifies a reference image frame including a replacement row corresponding to the overexposed row of the defective image frame. The defective image frame is modified by the data processing hardware by replacing the overexposed row with the corresponding replacement row from the reference image frame of the optical device stream. In another embodiment, the data processing hardware applies a temporal filter median filter to the overexposed row of the defective image frame.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the method further includes generating a row-exposure curve for each image frame using the data processing hardware. Here, the difference between the row exposure values of the neighboring rows is provided as a slope of the row-exposure curve.

In some examples, the reference frame is at least one of the image frames immediately preceding and immediately succeeding the defective image frame.

In some implementations, the row exposure value is a sum of a pixel values for all pixels in a row.

In some implementations, the optical device is a rolling shutter complementary metal oxide semiconductor sensor.

In some examples, the method includes determining, by the data processing hardware, that the defective image frame should be modified where a modification threshold is satisfied.

In some implementations, the overexposed row of the defective image frame corresponds to a period of an electromagnetic radiation pulse.

Another aspect of the disclosure provides a system including data processing hardware and memory hardware. The memory hardware is in communication with the data processing hardware, and stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. One of the operations may include receiving an optical device stream from an optical device. Another operation includes identifying image frames of the optical device stream, each of the image frames having an array of pixels including a plurality of rows. Additional operations can include determining a row exposure value for each of the rows in each of the image frames in the optical device stream, and then identifying a defective image frame including an overexposed row where a difference between the row exposure values of two neighboring rows of an image frame exceeds a threshold value. Another operation may include identifying a reference image frame including a replacement row corresponding to the overexposed row of the defective image frame. The defective image frame is modified by replacing the overexposed row with the corresponding replacement row from the reference image frame of the optical device stream.

This aspect may include one or more of the following optional features.

In some implementations, the method further includes generating a row-exposure curve for each image frame using the data processing hardware. Here, the difference between the row exposure values of the neighboring rows is provided as a slope of the row-exposure curve.

In some examples, the reference frame is at least one of the image frames immediately preceding and an immediately succeeding the defective image frame.

In some implementations, the row exposure value is a sum of the pixel values for all pixels in a row.

In some examples, the operations include applying a temporal filter median filter to the defective image frame.

In some implementations, the optical device is a rolling shutter complementary metal oxide semiconductor sensor.

In some examples, the method includes determining, by the data processing hardware, that the defective image frame should be modified where a modification threshold is satisfied.

In some implementations, the overexposed row of the defective image frame corresponds to a period of an electromagnetic radiation pulse.

The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims.

