CROSS REFERENCE TO RELATED PATENT APPLICATIONS

This application claims priority to the following Chinese Patent Applications: (1) CN201910177941.5, filed on 7 Mar. 2019, entitled “Method, Apparatus, Terminal, Capturing System, and Device for Setting Capturing Devices”, (2) CN201910172743.X, filed on 7 Mar. 2019, entitled “Method, Apparatus, Medium, and Device for Generating Multi-Angle Free-Perspective Image Data”, (3) CN201910172727.0, filed on 7 Mar. 2019, entitled “Method, Apparatus, Medium, and Server for Generating Multi-angle Free-perspective Video Data”, (4) CN201910172742.5, filed on 7 Mar. 2019, entitled “Method, Apparatus, Medium, Terminal, and Device for Processing Multi-Angle Free-Perspective Data”, (5) CN201910172729.X, filed on 7 Mar. 2019, entitled “Method, Apparatus, Medium, Terminal, and Device for Multi-Angle Free-Perspective Interaction”, (6) CN201910173415.1, filed on 7 Mar. 2019, entitled “Method, Apparatus, Medium, Terminal, and Device for Multi-Angle Free-Perspective Interaction”, (7) CN201910173413.2, filed on 7 Mar. 2019, entitled “Method, Apparatus, Medium, and Device for Processing Multi-Angle Free-Perspective Image Data”, (8) CN201910173414.7, filed on 7 Mar. 2019, entitled “Method, Apparatus, Medium, and Device for Processing Multi-Angle Free-Perspective Video Data”, (9) CN201910172761.8, filed on 7 Mar. 2019, entitled “Video Generating Method, Apparatus, Medium, and Terminal”, (10) CN201910172717.7, filed on 7 Mar. 2019, entitled “Video Reconstruction Method, System, Device, and Computer Readable Storage Medium”, (11) CN201910172720.9, filed on 7 Mar. 2019, entitled “Image Reconstruction Method, System, Device, and Computer-Readable Storage Medium”, which are all hereby incorporated by reference in their entirety.

TECHNICAL FIELD

Example embodiments of the present disclosure relate to the field of image processing technology, and in particular, to an image reconstruction method, system, device, and computer-readable storage medium.

BACKGROUND

With the continuous development of interconnection technologies, more and more video platforms continuously improve visual experience of user by providing videos with higher definition or viewing smoothness.

However, for videos with a strong sense of live experience, such as the image of a basketball game, the user often can only watch the game through one viewpoint position during viewing, and cannot switch the viewpoint position freely to view the game picture or game process at different perspective positions.

The 6 degrees of freedom (6DoF) technology is a technology for providing a high degree of freedom of the viewing experience. The user may adjust the perspective of viewing with interactive measures, so as to view from a desired free viewpoint angle. Thus, the viewing experience may be significantly enhanced. When the 6DoF image is viewed at any moment of the process, the user may choose to pause the video, so that the user may stay in this wonderful moment to have 6DoF free-perspective experience.

In order to realize 6DoF images, there are currently Free-D playback technology and light field rendering technology. The Free-D playback technology expresses 6DoF images through the point cloud. The point cloud expresses and stores the three-dimensional positions and pixel information of all points in space. The light field rendering technology establishes the light field database of the scenario through a set of pre-shot scenario photos under the condition that the content does not require depth information or association of the image. Then, for any given new viewpoint, the view of the viewpoint is obtained through resampling and bilinear interpolation operations, so as to achieve the roaming of the entire scenario.

However, both the Free-D playback technology and the light field rendering technology require a very large amount of data operation. In addition, currently, there is no good standard or industrial software and hardware supporting the compression of the point cloud, which is not conducive to be promoted and popularized.

SUMMARY

This Summary is provided to introduce a selection of implementations in a simplified form that are further described below in Detailed Description. This Summary is not intended to identify all features of the claimed subject matter, nor is it intended to be used alone as an aid in determining the scope of the claimed subject matter. The term “techniques,” for instance, may refer to device(s), system(s), method(s) and/or processor-readable/computer-readable instructions as permitted by the context above and throughout the present disclosure.

In view of the above, example embodiments of the present disclosure provide an image reconstruction method, system, device, and computer-readable storage medium to reduce the amount of data operation during the multi-degree-of-freedom image reconstruction.

In one aspect, example embodiments of the present disclosure provide an image reconstruction method. The method includes acquiring a multi-angle free-perspective image combination, parameter data of the image combination, and virtual viewpoint position information based on user interaction, wherein the image combination includes multiple groups of texture images and depth maps that are synchronized at multiple angles and have corresponding relationships; selecting a corresponding group of texture images and depth maps in the image combination at a user interaction moment based on a preset rule according to the virtual viewpoint position information and the parameter data of the image combination; and combining and rendering the selected corresponding group of texture images and depth maps in the image combination at the user interaction moment based on the virtual viewpoint position information and parameter data corresponding to the corresponding group of texture images and depth maps in the image combination at the user interaction moment, to obtain a reconstructed image corresponding to the virtual viewpoint position at the user interaction moment.

In an example embodiment, selecting the corresponding group of texture images and depth maps data in the image combination at the user interaction moment based on the preset rule according to the virtual viewpoint position information and the parameter data of the image combination includes: selecting the corresponding group of texture images and the depth maps in the image combination at the user interaction moment that satisfies a preset positional relationship and/or a preset quantitative relationship with the virtual viewpoint position according to the virtual viewpoint position information and the parameter data of the image combination.

In an example embodiment, selecting the corresponding group of texture images and the depth maps in the image combination at the user interaction moment that satisfies the preset positional relationship and/or the preset quantitative relationship with the virtual viewpoint position according to the virtual viewpoint position information and the parameter data of the image combination includes: selecting a preset number of corresponding groups of texture images and the depth maps that are closest to the virtual viewpoint position in the image combination at the user interaction moment according to the virtual viewpoint position information and the parameter data of the image combination.

In an example embodiment, combining and rendering the selected corresponding group of texture images and depth maps in the image combination at the user interaction moment based on the virtual viewpoint position information and parameter data corresponding to the corresponding group of texture images and depth maps in the image combination at the user interaction moment to obtain the reconstructed image corresponding to the virtual viewpoint position at the user interaction moment comprises: respectively projecting the depth maps of the corresponding group in the image combination at the user interaction moment to the virtual viewpoint position at the user interaction moment according to a spatial geometric relationship, to form a depth map of the virtual viewpoint position; and copying from pixel points in the texture images of the corresponding group to generated virtual texture images corresponding to the virtual viewpoint position according to the projected depth maps, to form the virtual texture images corresponding to the corresponding group in the image combination at the user interaction moment; and fusing the virtual texture images corresponding to the corresponding group in the image combination at the user interaction moment.

In an example embodiment, fusing the virtual texture images corresponding to the corresponding group in the image combination at the user interaction moment to obtain the reconstructed image of the virtual viewpoint position at the user interaction moment comprises: performing weighting processing on pixels in corresponding positions in the virtual texture images corresponding to respective corresponding groups in the image combination at the user interaction moment, to obtain pixel values of the corresponding positions in the reconstructed image of the virtual viewpoint position at the user interaction moment; and for a position where a pixel value is zero in the reconstructed image of the virtual viewpoint position at the user interaction moment, performing inpainting using pixels around the pixel in the reconstructed image, to obtain the reconstructed image of the virtual viewpoint position at the user interaction moment.

In an example embodiment, wherein fusing the virtual texture images corresponding to the corresponding group in the image combination at the user interaction moment to obtain the reconstructed image of the virtual viewpoint position at the user interaction moment comprises: for a position where a pixel value is zero in the virtual texture images corresponding to respective corresponding groups in the image combination at the user interaction moment, performing inpainting using around pixel values respectively; and performing weighting processing on pixel values in corresponding positions in the virtual texture images corresponding to the respective corresponding groups after the inpainting, to obtain the reconstructed image of the virtual viewpoint position at the user interaction moment.

On another aspect, example embodiments of the present disclosure provide an image reconstruction system. The system includes: an acquiring unit, adapted to acquire multiple-angle free-perspective image combination, parameter data of the image combination, and the virtual viewpoint position information based on user interaction, where the image combination includes multiple groups of texture images and the depth maps that are synchronized at multiple angles and have corresponding relationships; a selecting unit, adapted to select a corresponding group of texture images and the depth maps in the image combination at the user interaction moment based on a preset rule according to the virtual viewpoint position information and the parameter data of the image combination; and an image reconstruction unit, adapted to combine and render the selected corresponding group of texture images and depth maps in the image combination at the user interaction moment based on the virtual viewpoint position information and parameter data corresponding to the corresponding group of texture images and depth maps in the image combination at the user interaction moment, to obtain a reconstructed image corresponding to the virtual viewpoint position at the user interaction moment.

Example embodiments of the present disclosure further provide an image reconstruction device, including a memory and a processor. The memory stores computer instructions thereon that can run on the processor. When the processor runs the computer instructions, the processor performs the steps of the image reconstruction method according to any one of the above example embodiments.

In an example embodiment, the image reconstruction device includes at least one of a terminal device and an edge node.

Example embodiments of the present disclosure further provide a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed, steps of the image reconstruction method of any one of the above example embodiments are performed.

With the image reconstruction methods in the example embodiments of the present disclosure, by acquiring a multi-angle free-perspective image combination, parameter data of the image combination, and virtual viewpoint position information based on user interaction, wherein the image combination includes multiple groups of texture images and depth maps that are synchronized at multiple angles and have corresponding relationships; and selecting a corresponding group of texture images and depth maps in the image combination at a user interaction moment based on a preset rule according to the virtual viewpoint position information and the parameter data of the image combination; and next, it is only necessary to combine and render the selected corresponding group of texture images and depth maps in the image combination at the user interaction moment based on the virtual viewpoint position information and parameter data corresponding to the corresponding group of texture images and depth maps in the image combination at the user interaction moment. There is no need to perform image reconstruction based on the texture maps and depth maps of all groups in the image combination. Thus, the amount of data operation during the image reconstruction may be reduced.

Further, the corresponding group of texture images and the depth maps in the image combination at the user interaction moment that satisfies a preset positional relationship and/or a preset quantitative relationship with the virtual viewpoint position is selected according to the virtual viewpoint position information and the parameter data of the image combination. Thus, a higher degree of freedom and flexibility of selection may be provided in the case of reducing the amount of data operation and ensuring the quality of the reconstructed image. In addition, installation requirements for the shooting devices that capture images are also lowered, thereby facilitating the adaption to different site requirements and easy operation of installation.

Further, a preset number of corresponding groups of texture images and the depth maps that are closest to the virtual viewpoint position in the image combination at the user interaction moment is selected according to the virtual viewpoint position information and the parameter data of the image combination. Thus, the amount of data operation may be reduced and the reconstructed image quality may be guaranteed.

Further, texture images and depth maps corresponding to 2 to N capturing devices closest to the virtual viewpoint position are selected according to the virtual viewpoint position information and the parameter data corresponding to the corresponding group of texture images and depth maps in the image combination at the user interaction moment, where N is the number of all capturing devices that capture the image combination. Thus, texture images and corresponding depth maps captured by multiple capturing devices that are closest to the virtual viewpoint position may be selected according to needs. Thus, the reconstructed image that satisfies the definition requirements may be obtained with as little data as possible. Meanwhile, transmission resources may also be saved.

Further, performing forward projection on the selected depth maps of the corresponding group in the image combination at the user interaction moment respectively, to project to the virtual position at the user interaction moment; performing post-processing on the depth maps after the forward projection respectively; next, performing backward projection on the selected texture images of the corresponding group in the image combination at the user interaction moment respectively; and fusing respective virtual texture images generated after the backward projection. After the above processing, the reconstructed image of the image combination at the user interaction moment. Because merely some of the texture images and depth maps in the image combination are used to perform the reconstruction, and the used multi-angle free-perspective image data of the image combination including texture images and depth maps occupies a relatively low data amount, processing resources may be saved and video reconstruction efficiency may be improved.

Further, inpainting is performed on the fused texture image to obtain the reconstructed image corresponding to the virtual viewpoint position at the user interaction moment. The reconstructed image quality may be improved.

Further, during the post-processing of the depth map after the forward projection, by performing foreground padding processing on the depth maps after the forward projection respectively, and/or performing pixel-level filtering processing on the depth maps after the forward projection respectively, the quality of the reconstructed depth map may be improved.

Further, fusing the respective virtual texture images generated after the backward projection using a global weight determined by a distance between the virtual viewpoint position and a position of a capturing device that captures a corresponding texture image in the image combination, according to the virtual viewpoint position information and the parameter data corresponding to the corresponding group of texture images and depth maps in the image combination at the user interaction moment, may make the reconstructed image more real, and further improve the quality of the reconstructed image.

Further, by performing the image reconstruction using the methods of the example embodiments of the disclosure after decoding acquired compressed multi-angle free-perspective image data, network transmission resources may be further saved. Moreover, the compression may also be performed in the general compression manner with the compression software and hardware. Thus, the method is conducive to be promoted and popularized.

Further, the image reconstruction solutions in the example embodiments of the present disclosure are applied to devices such as terminal devices, edge node devices, and the like, and may be adapted to devices with limited computing capabilities such as terminal devices, edge node devices, and the like, to satisfy the needs of the user for viewing images based on the virtual viewpoint, and enhance the user's visual experience.

