In recent years, we have greeted the age of multimedia in which audio, video, and other data are integrally handled, and the conventional information media, i.e., means for transmitting information between persons, such as newspapers, magazines, televisions, radios, and telephones, have been grasped as subjects of multimedia. Generally, “multimedia” does not mean only representing characters, but means representing especially images, simultaneously in relation with diagrams, speech, and the like. In order to adopt conventional information as information media as subjects of multimedia, it is required to represent the information in a digital form.
When the information quantity possessed by the respective information media described above are estimated as digital information quantity, the information quantity per character is 1-2 bytes, while the information quantity of more than 64 kbits per second (telecommunication quality) is required for audio and the information quantity of more than 100 Mbits per second (current television broadcasting quality) is required for moving picture. Therefore, it is not practical to handle such tremendous amounts of data of the above-described information media as they are in digital formats. For example, though visual phones have already been put to practical use by ISDN (Integrated Services Digital Network) having a transmission rate of 64 kbps˜1.5 Mbps, it is impossible to transmit images of television cameras as they are by the ISDN.
In these circumstances, information compression techniques are required. For example, in the case of visual telephones, the moving picture compression techniques standardized as H.261 and H.263 by ITU-T (International Telecommunication Union-Telecommunication Sector) are employed. Further, according to the information compression technique of MPEG1, it is possible to contain video information together with audio information in an ordinary music CD (compact disk).
MPEG (Moving Picture Experts Group) is an international standard of data compression for a moving picture, i.e., pixel values of a moving picture. MPEG1 is a standard for compressing pixel values of a moving picture to 1.5 Mbps, i.e., data of a television signal to about 1/100. Further, while the transmission rate in MPEG1 is mainly limited to about 1.5 Mbps, in MPEG2 which is aimed at standardization to meet a request for a higher image quality, the limitation is relaxed with pixel values of a moving picture being compressed to 2˜15 Mbps.
Further, under the existing circumstances, standardization of MPEG4 has been almost completed by the working group for standardization of MPEG1 and MPEG2 (ISO/IEC JTC1/SC29/WG11), which enables coding and handling in object units and realizes new functions demanded in the multimedia age. While MPEG4 has initially aimed at standardization of a coding method of a low bit rate, the aim of the standardization thereof is now extended to a more versatile coding process of a high bit rate or an interlaced image. One of characteristics of MPEG4 is coding simultaneously plural image sequences and transmitting the same. This enables one image scene to be composed of plural images. The foreground and the background can be different image sequences, and the frame frequency, the image quality and the bit rate thereof can be individually changed. Thereby, plural images can be arranged in the horizontal or vertical direction like in a multi-screen, and it is enabled for the user to extract or enlarging-display only a desired image. It is general that only pixel values are coded for the background similarly in MPEG2, while, as for the foreground, a pixel value signal indicating pixel values of the object as well as a shape signal indicating the shape of the object are coded. Commonly, the coding of the foreground are known as coding in object units. The displayed image is a video composed of the respective decoded images.
FIGS. 8(a)-8(f) are diagrams for explaining video composition in object units. FIG. 8(a) shows pixel values of a foreground video (hereinafter, simply referred to as foreground) fv1 of a balloon which composes a display video. FIG. 8(b) shows a shape value of a shape signal fml corresponding to the foreground fv1 in FIG. 8(a). FIG. 8(c) shows pixel values of a foreground fv2 of a human which composes the display video. FIG. 8(d) shows a shape value of a shape signal fm2 corresponding to the foreground fv2 in FIG. 8(c). FIG. 8(e) shows pixel values of a background video (hereinafter, simply referred to as background) rv. FIG. 8(f) shows the display video which is obtained by composing the foregrounds fv1 and fv2 of FIGS. 8(a) and 8(c) with the background rv of FIG. 8(e).
In the screens of the shape signals of FIGS. 8(b) and 8(d), parts painted black in the screens show areas in which the corresponding pixel values exist, i.e., inside an object, and white parts show areas in which no corresponding pixel values exist, i.e., outside the object. A wording that “a pixel value is significant/insignificant” is sometimes used in a sense that a pixel value exists/no pixel value exists.
FIG. 9 is a block diagram illustrating a structure of a prior art video decoding system. In this figure, reference DeMux denotes a stream demultiplexer for demultiplexing a multiplexed stream StrM. References Dec1, Dec2 and Dec3 denote decoders for decoding video streams Str1, Str2 and Str3 which have been demultiplexed by the stream demultiplexer DeMux, respectively. Reference Comp1 denotes a video composer for composing decoded videos Dout1, Dout2 and Dout3 which have been decoded by the decoders Dec1, Dec2 and Dec3, respectively. Reference Disp denotes a display unit for displaying a video Vcom which is composed by the video composer Comp1. Reference IF denotes an object selector for the user to select an object. CPU denotes a controller for controlling the decoders Dec1, Dec2 and Dec3 in accordance with the instruction of the object selector IF.
Next, the operation of the prior art video decoding system is described. A multiplexed stream StrM is demultiplexed by the stream demultiplexer DeMux into video streams Str1, Str2 and Str3 corresponding to videos of three objects, as well as overlap information Odr indicating the order of overlap of these videos which is sent to the controller CPU. The decoder Dec1 decodes the video stream Str1 and outputs the decoded video Dout1. Similarly, the decoders Dec2 and Dec3 decode the video streams Str2 and Str3, and output the decoded videos Dout2 and Dout3, respectively. The video composer Comp1 composes these decoded videos Dout1, Dout2 and Dout3 to provide a composed video Vcom, and displays the composed video on the display unit Disp.
On the other hand, in the case of object unit coding, the user can switch the display/non-display of videos in object units. The user selects the display or non-display of each object by means of the object selector IF. The object selector IF notifies the video composer Comp1 of object non-display information Dsel according to this selection, and the video composer Comp1 composes only videos of objects which should be displayed, to display the composed video.
With using the shape value of an object, it can be judged whether a position is inside or outside the object. Accordingly, by executing an operation of selecting a button Bn within the screen using a pointer Pr as shown in FIG. 10, it is possible that the user designates a specific position in the screen and obtains information as to whether the position is inside or outside an object. To be specific, the user moves the pointer by the object selector IF and selects a button, thereby notifying the controller CPU of object selection information Psel which indicates a designated pixel position. The controller CPU makes an inquiry to the decoders Dec1, Dec2 and Dec3 corresponding to the respective objects with object judge commands Q1, Q2 and Q3 about whether the designated position is inside or outside an object such as a button on the screen. The decoders Dec1, Dec2 and Dec3 report the controller CPU whether the inquired position is inside or outside the object by object judge results A1, A2 and A3, respectively, and then the controller CPU notifies the user or applications of the object judge results A1, A2 and A3 collectively as object judge result Req.
The block diagram of FIG. 9 illustrating the video decoding system shows an example where three videos are composed to obtain a composed video Vcom, while the number of videos to be composed can be less than three, or more than three. In addition, in this example, one decoder is provided for each video stream, while when plural video streams can be decoded by one video decoder by time division or the like, the number of video decoders can be properly reduced.
FIG. 11 is a block diagram illustrating a structure of the decoder Dec of the video decoding apparatus in the prior art video decoding system shown in FIG. 9. In this figure, a video stream Str, an object judge command Q, an object judge result A and a decoded video Dout correspond to one of the video streams Str1, Str2 and Str3, the object judge commands Q1, Q2 and Q3, the object judge results A1, A2 and A3, and the decoded videos Dout1, Dout2 and Dout3 in FIG. 9, respectively. Reference DecU denotes a video decoding unit for decoding the video stream Str. References MEM1, MEM2, MEM3 and MEM4 denote memories for containing decoded videos mem1, mem2, mem3 and mem4, respectively.
Next, the operation of the decoder Dec is described. In FIG. 11, the video decoding unit DecU decodes the video stream Str and stores the decoded videos mem1, mem2, mem3 and mem4 which are obtained by the decoding, into the memories MEM1, MEM2, MEM3 and MEM4, respectively. At this time, when the video stream Str has been inter-frame coded, the decoded videos mem1, mem2, mem3 and mem4 are read from the memories MEM1, MEM2, MEM3 and MEM4 to utilize the same as reference videos at the motion compensation. Since a signal of a video having the shape is composed of three components indicating the color (YUV, RGB or the like) and a shape value A, i.e., four components in total, the four individual memories are provided to correspond to the four components, respectively. However, when practically packaged, these can be integrated in one memory.
In the example shown in FIG. 11, the memories MEM1, MEM2, MEM3 and MEM4 contain a luminance pixel value Yimg, two color difference pixel values Uimg and Vimg, and a shape value Aimg, respectively. Since the shape value Aimg is stored in the memory MEM4, when a pixel position such as a position pointed by the pointer is indicated by the object judge command Q from the controller CPU in FIG. 9, the memory MEM4 judges whether that position is inside or outside the object, and outputs the result as the object judge result A. The decoded videos stored in the memories MEM1, MEM2, MEM3, and MEM4 are read as pixel decoded videos Yimg, Uimg and Vimg, and a shape decoded video Aimg at a timing of display, to obtain pixel decoded videos Yout, Uout and Vout, and a shape decoded video Aout, respectively. The decoded video Dout is obtained by combining the pixel decoded videos Yout, Uout and Vout and the shape decoded video Aout.
FIG. 12 is a diagram showing an internal structure of the memory MEM4 which contains the shape value. In this figure, reference MEM41 denotes a shape signal storage memory for containing the shape signal. Reference CMP denotes a pointed position comparison means for comparing and judging whether the pointer operated by the object selector IF in FIG. 9 points inside or outside of an object, such as a button.
Next, the operation of the memory MEM4 is described. In FIG. 12, the shape signal storage memory MEM41 contains the shape signal mem4 which has been decoded by the decoding unit DecU in FIG. 11 as a bitmap. The pointed position comparison means CMP converts pointed position information of the pointer operated by the object selector IF in FIG. 9, which is transmitted in accordance with the object judge command Q issued by the controller CPU in FIG. 11, together with this object judge command Q, into an address of the shape signal storage memory MEM41, and judges whether or not the bitmap of the shape signal exists at that address, thereby judging whether the position pointed by the pointer is inside or outside of the video of the object, such as a button. Then, the pointed position comparison means CMP outputs the judge result to the controller CPU in FIG. 9 as the object judge result A.
As described above, the information as to whether a position is inside or outside an object can be obtained by using the shape value of the object. By utilizing these workings, the shape value can be used as a GUI (Graphic User Interface) operable button whose shape varies. This is what is called a “hot spot”, and, for example, when a certain position on the screen of a terminal (for example, assuming that this is a video in a form of a button) is clicked to make the terminal execute a special processing, a shape signal indicating that position is transmitted as a moving picture, thereby changing the position or shape of the button. For that purpose, not only videos including both of pixel values and shape values, but also videos having only shape values are effective, and accordingly the coding of only shape values can be also used in MPEG4.
For example, when FIG. 8(f) is used as background, only shape values of FIGS. 8(b) and 8(d) are coded, and a position is pointed in FIG. 8(f) by a pointer, the judgement as to whether that position is inside or outside the human or balloon can be made. Therefore, when only the judgement as to whether a position is inside or outside the video is to be made, the coding of individual pixel values of FIGS. 8(a), 8(c) and 8(e) is not required, and only the coding of the pixel value of FIG. 8(f) is required. Therefore, the coding/decoding process can be simplified, and in some cases the compression rate can be also increased by reduction of the number of pixels to be coded.
From the above descriptions, it can be seen that there are three kinds of the stream of color videos (moving pictures) in MPEG4, i.e., only YUV (color signals) in the case of videos whose shapes do not vary, YUV+A (a shape signal is added to the color signals) in the case where coding in object units is carried out, and only A (shape signal) in the case of judgement as to whether a position is inside or outside a video is made.
There are some cases where videos are transmitted according to MPEG1 and a shape signal according to MEPG4 is added thereto. Further, there is also a case where a texture is pasted on a shape signal transmitted according to MPEG4, whereby MPEG4 and CG (Computer Graphics) are combined to display color videos.
FIG. 13 is a diagram schematically showing the format of a video stream Str corresponding to a video of an object. In FIG. 13, reference HD denotes the header of the entire stream. References DA1, . . . , and DAX denote data of one screen, respectively. References HD1, . . . , and HDX denote headers corresponding to the screen data DA1, . . . , and DAX, respectively.
In the header HD of the entire stream, the video size (when the size of the video does not vary with frames) and the coding method (quantization method or information relating to arrangement of data) as well as information indicating a target which is being coded (above-mentioned YUV, YUV+A, A or the like), are coded and stored.
In the headers HD1, . . . , and HDX corresponding to the screen data DA1, . . . , and DAX, respectively, parameters required for the decoding, information indicating which frame is the corresponding video data or which is of I frame and P frame the video data, and the like are coded and stored.
FIG. 14 is a diagram showing a structure of a multiplexed stream StrM which is obtained by multiplexing plural video streams Str each corresponding to a video of an object. In the example shown in FIG. 14, the video streams Str are time-divided multiplexed frame by frame, and a header MHD including the overlap information Odr is arranged between the video streams Str.
As described above, it is useful to code only shape values, while when a stream including no pixel value but having only the shape value is received, what becomes the pixel value which is obtained by decoding this stream is not decided in the MPEG4 standard at the present time.
Originally, a stream having only the shape value is created provided that it is not displayed on the receiving end. However, since the measures to be taken when this is received and decoded are not defined in the MPEG4 standard, the stream having only the shape value should not be displayed, in accordance with proposals on the application side which utilizes video communication to provide information terminals with various kinds of services.
However, in many cases, video decoders created for general purposes are generally used for various applications to reduce developing costs, and these video decoders are designed to always decode and display transmitted information. Therefore, also when receiving a stream having only the shape value, the video decoder decodes this stream, and some pixel values which cannot be predicted are displayed due to that decoding, thereby giving wrong or unpleasant feelings to persons who watch the screen.