RELATED APPLICATION

This application was originally filed as PCT Application No. PCT/FI2018/050581, filed on Aug. 13, 2018, which claims priority from FI Application No. 20175731, filed on Aug. 15, 2017.

BACKGROUND

Volumetric video data represents a three-dimensional scene or object and can be used as input for virtual reality (VR), augmented reality (AR) and mixed reality (MR) applications. Such data describes the geometry, e.g. shape, size, position in 3D-space, and respective attributes, e.g. color, opacity, reflectance and any possible temporal changes of the geometry and attributes at given time instances, comparable to frames in 2D video. Volumetric video is either generated from 3D models through computer-generated imagery (CGI), or captured from real-world scenes using a variety of capture solutions, e.g. multi-camera, laser scan, combination of video and dedicated depth sensors, and more. Also, a combination of CGI and real-world data is possible.

Typical representation formats for such volumetric data are triangle meshes, point clouds (PCs), or voxel arrays. Temporal information about the scene can be included in the form of individual capture instances, i.e. “frames” in 2D video, or other means, e.g. position of an object as a function of time.

The above mentioned volumetric video representation formats suffer from poor spatial and temporal coding performance.

There is, therefore, a need for solutions for improved coding of volumetric video.

SUMMARY

Now there has been invented an improved method and technical equipment implementing the method, by which the above problems are alleviated. Various aspects of the invention include a method, an apparatus (an encoder and a decoder), a system and a computer readable medium comprising a computer program stored therein, which are characterized by what is stated in the independent claims. Various details of the invention are disclosed in the dependent claims and in the corresponding images and description.

A volumetric video, for example dynamic point clouds, arrays of voxels or mesh models or a combination of such, may be projected onto a number of projection surfaces having simple geometries, for example sphere(s), cylinder(s), cube(s), polyhedron(s) and/or plane(s). In this context, a projection surface may be a piece-wise continuous and smooth surface in three-dimensional space. Piece-wise smoothness may be understood so that there are regions of the surface where the direction of the surface normal does not change abruptly (i.e. the values of the coefficients of the surface normal's coordinate components are continuous). A projection surface may comprise pieces of simple geometric surfaces. A projection surface may also evolve (change) over time. On such surfaces, the texture and geometry of point clouds, voxel arrays or mesh models may form pixel images, e.g. texture images and depth images (indicative of distance from the projection plane). Such projection surfaces may be unfolded onto two-dimensional (2D) planes, e.g. resulting in a two-dimensional pixel image. Standard 2D video coding may be applied for each projection to code the pixel information resulting from the texture data. In connection with the texture information, relevant projection geometry information, i.e. comprising e.g. projection or projection surface type, location and orientation of the projection surface in 3D space and/or size of the projection surface, may be transmitted either in the same bitstream or separately along the bitstream. At the receiver side, the bitstream may be decoded and volumetric video may be reconstructed from decoded 2D projections and projection geometry information.

In an encoding method, a bitstream may be encoded or encapsulated, the bitstream comprising one or more coded texture pictures and one or more coded accompanying additional pictures per texture picture. Each texture picture may represent a 2D projection of 3D data onto a given geometry. Several projections may be frame-packed into a single picture. An additional geometry picture may represent geometry data, e.g. depth. Additional auxiliary pictures may represent further attributes, e.g. surface normal, reflectance and opacity, transmittance and/or refraction of the 3D data. Relevant projection information may be indicated in or along the bitstream. For example, general projection information may be given in or along the bitstream: number of coded texture pictures or frame-packing information. Projection geometry information may be given for the projections: for example comprising shape, size, location and orientation of the projection surfaces. Temporal changes in location, orientation and size of such geometries may be encoded, possibly as function of time. Possible further sub-division of such geometries and resulting changes in geometry information may be given. Nature of auxiliary geometry data may be given: bit depth, quantisation, value range and/or inverse values, that is, coding of negative radius or depth values of the geometry picture. Nature of other auxiliary data, i.e. surface normal, reflectance and opacity, transmittance and/or refraction, etc. may be given.

In a decoding method, a bitstream comprising one or more coded texture picture(s) and one or more coded accompanying geometry pictures and auxiliary pictures per texture picture may be received and decoded. The texture picture represents a two-dimensional projection of three-dimensional data onto a projection surface with a given geometry. A geometry picture may represent geometry data, e.g. depth from the projection surface. Additional auxiliary pictures may represent further attributes, e.g. surface normal, reflectance, opacity, etc. as described earlier. Relevant projection information may also be decoded from or along a bitstream. From this decoded information a 3D scene model may be reconstructed accordingly.

A 3D object, represented as point cloud, may be sequentially projected onto a projection surface, i.e. the 3D projection is performed from different directions of the 3D object, one after another. In the case of occlusions of a certain primitive (point) when two points in 3D space are mapped on the same pixel of the projection surface, only the outmost points of the object surface are projected onto the projection surface. Successfully projected primitives (points) are then removed from the 3D object and the process is repeated for the next direction of projection. Step-by-step, the number of remaining points per projection will decrease, thus occlusions and mapping errors will be reduced. The process can be repeated until all sides of a geometry are covered, or a sufficiently large number of points has been projected (number of projection steps can be smaller, equal or larger than sides or natural directions of the projection surface).

As the numbers of projected points decrease, the projected 2D planes become more and more sparse. The coding efficiency of such sparse content may be increased by inpainting techniques to reduce sparsity. To avoid inpainted pixels creating new points in the reconstructed 3D object, masking or prediction-based determination of inpainted points at the decoder may be used.

According to a first aspect, there is provided a method, comprising: encoding a first texture picture into a bitstream, said first texture picture comprising a first projection of first volumetric texture data of a first source volume of a scene model and a second projection of second volumetric texture data of said first source volume of said scene model, said first projection being from said first source volume to a first projection surface, and said second projection being from said first source volume to a second projection surface, said second volumetric texture data having been obtained by removing at least a part of said first volumetric texture data that has been successfully projected in said first projection, encoding a first geometry picture into said bitstream, said geometry picture representing a mapping of said first projection surface to said first source volume and a mapping of said second projection surface to said first source volume, and encoding projection geometry information of said first and second projections into said bitstream, said projection geometry information comprising information of position of said first and second projection surfaces in said scene model.

According to a second aspect, there is provided a method, comprising: decoding a first texture picture from a bitstream to obtain first and second decoded texture data of a source volume, said first texture picture comprising a first projection of first volumetric texture data of a first source volume of a scene model and a second projection of second volumetric texture data of said first source volume of said scene model, said first projection being from said first source volume to a first projection surface, and said second projection being from said first source volume to a second projection surface, said second volumetric texture data having been obtained by removing at least a part of said first volumetric texture data that has been successfully projected in said first projection, decoding a first geometry picture from said bitstream to obtain first and second decoded geometry data of a source volume, said geometry picture representing a mapping of said first projection surface to said first source volume and a mapping of said second projection surface to said first source volume, decoding projection geometry information of said first and second projections from said bitstream, said projection geometry information comprising information of position of said first and second projection surfaces in said scene model, and forming a reconstructed scene model by projecting said first decoded texture data to a first destination volume using said first decoded geometry data and said first projection geometry information and by projecting said second decoded texture data to said first destination volume using said second decoded geometry data and said projection geometry information.

According to a third aspect, there is provided an apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following: encode a first texture picture into a bitstream, said first texture picture comprising a first projection of first volumetric texture data of a first source volume of a scene model and a second projection of second volumetric texture data of said first source volume of said scene model, said first projection being from said first source volume to a first projection surface, and said second projection being from said first source volume to a second projection surface, said second volumetric texture data having been obtained by removing at least a part of said first volumetric texture data that has been successfully projected in said first projection, encode a first geometry picture into said bitstream, said geometry picture representing a mapping of said first projection surface to said first source volume and a mapping of said second projection surface to said first source volume, and encode projection geometry information of said first and second projections into said bitstream, said projection geometry information comprising information of position of said first and second projection surfaces in said scene model.

According to a fourth aspect, there is provided an apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following: decode a first texture picture from a bitstream to obtain first and second decoded texture data of a source volume, said first texture picture comprising a first projection of first volumetric texture data of a first source volume of a scene model and a second projection of second volumetric texture data of said first source volume of said scene model, said first projection being from said first source volume to a first projection surface, and said second projection being from said first source volume to a second projection surface, said second volumetric texture data having been obtained by removing at least a part of said first volumetric texture data that has been successfully projected in said first projection, decode a first geometry picture from said bitstream to obtain first and second decoded geometry data of a source volume, said geometry picture representing a mapping of said first projection surface to said first source volume and a mapping of said second projection surface to said first source volume, decode projection geometry information of said first and second projections from said bitstream, said projection geometry information comprising information of position of said first and second projection surfaces in said scene model, and form a reconstructed scene model by projecting said first decoded texture data to a first destination volume using said first decoded geometry data and said first projection geometry information and by projecting said second decoded texture data to said first destination volume using said second decoded geometry data and said projection geometry information.

According to a fifth aspect, there is provided a system comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the system to perform at least the following: encode a first texture picture into a bitstream, said first texture picture comprising a first projection of first volumetric texture data of a first source volume of a scene model and a second projection of second volumetric texture data of said first source volume of said scene model, said first projection being from said first source volume to a first projection surface, and said second projection being from said first source volume to a second projection surface, said second volumetric texture data having been obtained by removing at least a part of said first volumetric texture data that has been successfully projected in said first projection, encode a first geometry picture into said bitstream, said geometry picture representing a mapping of said first projection surface to said first source volume and a mapping of said second projection surface to said first source volume, encode projection geometry information of said first and second projections into said bitstream, said projection geometry information comprising information of position of said first and second projection surfaces in said scene model, decode said first texture picture from said bitstream to obtain first and second decoded texture data of a source volume, decode said first geometry picture from said bitstream to obtain first and second decoded geometry data of a source volume, decode said projection geometry information of said first and second projections from said bitstream, and form a reconstructed scene model by projecting said first decoded texture data to a first destination volume using said first decoded geometry data and said first projection geometry information and by projecting said second decoded texture data to said first destination volume using said second decoded geometry data and said first projection geometry information.

According to further aspects, there are provided computer program products embodied on a non-transitory computer readable medium, comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to carry out the method according to the first or second aspect.

