The method, apparatus, and system according to the present invention are configured to compute interpolated image data of a video image data by means of line-based motion estimation and compensation and to detect and handle errors in interpolated image data obtained as result of performing the line-based motion compensation. The present invention allows efficient use of chip-internal memory and efficient interacting of components, devices, and/or modules enabling the line-based motion estimation and compensation, and processing of the interpolated image data obtained as result of performing the line-based motion compensation, wherein the quality of the resulting image data to be visualized is improved considerably and in an effective way at the same time.
Hereinafter, the present invention and its underlying problem is described with regard to the processing of a video signal for line-based motion estimation and motion compensation within a video processing apparatus such as a microprocessor or microcontroller having line memory devices, whereas, it should be noted, that the present invention is not restricted to this application, but can also be used for other video processing apparatus.
The market introduction of TV-sets based on 100/120 Hz frame rate or even higher required the development of reliable Field/Frame Rate Up-conversion (FRU) techniques to remove artefacts within a picture such as large area flickers and line flickers. Standard FRU methods, which interpolate the missing image fields to be displayed on Displays without performing an estimation and compensation of the motion of moving objects in successive image fields, are satisfactory in many applications, especially with regard to a better quality of the image and with regard to the reduction of the above-mentioned artefacts. However, many pictures contain moving objects, like persons, subtitles and the like, which cause so-called motion judders.
This problem is better understood by referring to FIG. 1, wherein the motion trajectory of the moving objects (white squares) in the original image fields (i.e. transmitted and received image fields) is supposed to be straight-lined. If the missing fields/frames result from interpolation by means of the above mentioned standard FRU methods (i.e. without motion estimation and compensation), the motion of the moving object in the interpolated fields (dark grey squares) is not at a position as expected by the observer (dotted squares). Such artefacts are visible and induce a blurring effect especially of fast moving objects. These blurring effects typically reduce the quality of the displayed images significantly.
In order to avoid such blurring effects and to reduce artefacts several methods for motion estimation and motion compensation—or shortly MEMC—are proposed. This MEMC provides the detecting of a moving part or object within the received image fields and then the interpolation of the missing fields according to the estimated motion by incorporating the missing object or part in an estimated field.
FIG. 2 shows schematically the change of the position of a moving object between two successive image fields. Between two successive received image fields/frames, the moving objects will have changed their position, e. g. object MO which is in the previous field/frame T in position A is then in the current field/frame T+1 then in position B. This means, that a motion exists from the previous field/frame T to the current field/frame T+1. This motion of an object in successive image fields/frames can be represented by a so-called motion vector. The motion vector AB represents the motion of the object MO from position A in the previous field T to position B in the current field/frame T+1. This motion vector AB typically has a horizontal and a vertical vector component. Starting from point A in the previous field T and applying this motion vector AB to the object MO the object MO is then translated in position B in the current field/frame T+1. The missing position I of the object MO in the missing field/frame T+½ that has to be interpolated must be calculated by the interpolation of the previous field T and the current field T+1 taken account of the respective positions A, B of the moving object MO. If the object MO does not change its position between the previous field/frame and the current field/frame, e. g., if A and B are the same, position I in the missing field is obtained by the translation of A with a motion vector |AB|/2. In this manner the missing field T+½ is interpolated with a moving object in the right position with the consequence that blurring effects are effectively avoided.
Theoretically, for each pixel of a field a corresponding motion vector has to be calculated. However, this would increase the number of calculation needed and thus the memory requirements enormously. To reduce this enormous calculation and memory effort there exist basically two different approaches:
The first approach employs a so-called block-based MEMC. This first approach assumes that the dimension of the object in the image is always larger than that of a single pixel. Therefore, the image field is divided into several image blocks. For MEMC only one motion vector is calculated for each block.
The second approach employs a so-called line-based MEMC. In this second approach the algorithm is based on a reduced set of video input data of a single line of a field or a part of this line. The present invention is based on this second MEMC approach.
In present line-based MEMC systems, image data is usually stored in a local buffer or on chip memory, the so-called line memory, to which rather extreme bandwidth requirements are made. Many present MEMC systems, like the implementations described by Gerard de Haan in EP 765 572 B1 and U.S. Pat. No. 6,034,734, apply a cache memory (e.g. a two-dimensional buffer) to reduce the bandwidth requirements and to store a sub-set of an image. The motion compensation device or module fetches video image data from this cache while applying motion vectors. Typically, in MEMC systems this cache covers the whole search range of the motion vectors. Usually, the cache consists of a great amount of so-called line memories. This results in a relatively large amount of memory, e.g. 720 pixels wide and 24 lines (with an associated maximum vertical vector range of [−12−+12]. Such a cache comprising a great amount of single line memories requires a huge memory needed only for MEMC data buffering. As a consequence, the memory portion within the processor covers a relatively sizable chip area.
Commonly used MEMC algorithms compensate the motion in two directions, i.e. the motion in the horizontal direction and as well in the vertical direction. For that operation a memory access should be randomly possible, which requires for an application in hardware sufficient embedded chip memory within the video processor for the different temporal incoming data streams. The size of this embedded chip memory strongly depends on the search range (i.e. search area) for the motion of an object, as already outlined above, where the motion estimation can match similar video patterns in two temporal positions and derive the velocity of the motion in terms of pixels per frame or per field.
However, this matching process does not always work perfectly, since methods to determine the quality of the measured motion vector are required. Therefore, for the internal storage of further temporal incoming video signals additional memory resources are required. This, however, increases the amount of embedded memory even further, which leads to an increase of the chip area since for an integrated circuit it is the chip internal memory which significantly determines the chip area. Consequently, the chip is getting more and more expensive. Especially in the mainstream market segment such as for modern Plasma- and LCD-TVs these additional costs typically form a limiting factor for an MEMC implementation.
The present invention is, therefore, based on the object to provide a more efficient use of the chip-internal resources and especially of the chip-internal memory with regard to motion estimation and motion compensation, wherein the quality of the resulting image data is to be improved at the same time.