BACKGROUND

Technical Field

This disclosure relates generally to compression and decompression of point clouds comprising a plurality of points, each having associated spatial information and attribute information.

Description of the Related Art

Various types of sensors, such as light detection and ranging (LIDAR) systems, 3-D-cameras, 3-D scanners, etc. may capture data indicating positions of points in three dimensional space, for example positions in the X, Y, and Z planes. Also, such systems may further capture attribute information in addition to spatial information for the respective points, such as color information (e.g. RGB values), texture information, intensity attributes, reflectivity attributes, motion related attributes, modality attributes, or various other attributes. In some circumstances, additional attributes may be assigned to the respective points, such as a time-stamp when the point was captured. Points captured by such sensors may make up a “point cloud” comprising a set of points each having associated spatial information and one or more associated attributes. In some circumstances, a point cloud may include thousands of points, hundreds of thousands of points, millions of points, or even more points. Also, in some circumstances, point clouds may be generated, for example in software, as opposed to being captured by one or more sensors. In either case, such point clouds may include large amounts of data and may be costly and time-consuming to store and transmit.

SUMMARY OF EMBODIMENTS

In some embodiments, a system includes one or more sensors configured to capture points that collectively make up a point cloud, wherein each of the points comprises spatial information identifying a spatial location of the respective point and attribute information defining one or more attributes associated with the respective point.

The system also includes an encoder configured to compress the attribute and/or spatial information of the points. To compress the attribute and/or spatial information, the encoder is configured to determine, for the point cloud, a plurality of patches, each corresponding to portions of the point cloud, wherein each patch comprises points with surface normal vectors that deviate from one another less than a threshold amount. The encoder is further configured to, for each patch, generate a patch image comprising the set of points corresponding to the patch projected onto a patch plane and generate another patch image comprising d information, such as depth information, for the set of points corresponding to the patch, wherein the geometry information comprises depths of the points in a direction perpendicular to the patch plane.

For example, the geometry patch image corresponding to the patch projected onto a patch plane may depict the points of the point cloud included in the patch in two directions, such as an X and Y direction. The points of the point cloud may be projected onto a patch plane approximately perpendicular to a normal vector, normal to a surface of the point cloud at the location of the patch. Also, for example, the geometry patch image comprising depth information for the set of points included in the patch may depict depth information, such as depth distances in a Z direction. To depict the depth information, the geometry patch image may include a parameter that varies in intensity based on the depth of points in the point cloud at a particular location in the patch image. For example, the geometry patch image depicting depth information may have a same shape as the attribute patch image representing attributes of points projected onto the patch plane. However, the geometry information patch image may be an image comprising image attributes, such as one or more colors, that vary in intensity based on depth, wherein the intensity of the one or more image attributes corresponds to a depth of a corresponding point of the point cloud at a location in the geometry patch image where the image attribute is displayed in the geometry patch image depicting depth. For example, points that are closer to the patch plane may be encoded as darker values in the patch image depicting depth and points that are further away from the patch plane may be encoded as lighter values in the patch image depicting depth, for example in a monochromatic patch image depicting depth. Thus, the depth information patch image when aligned with other patch images representing attribute values for points projected onto the patch plane may indicate the relative depths of the points projected onto the patch plane, based on respective image attribute intensities at locations in the geometry patch image that correspond to locations of the points in the other patch images comprising point cloud points projected onto the patch plane.

The encoder is further configured to pack generated patch images (including a depth patch image and, optionally, one or more additional patch images for one or more other attributes) for each of the determined patches into one or more image frames and encode the one or more image frames. In some embodiments, the encoder may utilize various image or video encoding techniques to encode the one or more image frames. For example, the encoder may utilize a video encoder in accordance with the High Efficiency Video Coding (HEVC/H.265) standard or other suitable standards such as, the Advanced Video Coding (AVC/H.265) standard, the AOMedia Video 1 (AV1) video coding format produced by the Alliance for Open Media (AOM), etc. In some embodiments, the encoder may utilize an image encoder in accordance with a Motion Picture Experts Group (MPEG), a Joint Photography Experts Group (JPEG) standard, an International Telecommunication Union-Telecommunication standard (e.g. ITU-T standard), etc.

In some embodiments, a decoder is configured to receive one or more encoded image frames comprising patch images for a plurality of patches of a compressed point cloud, wherein, for each patch, the one or more encoded image frames comprise: a patch image comprising a set of points of the patch projected onto a patch plane and a patch image comprising depth information for the set of points of the patch, wherein the depth information indicates depths of the points of the patch in a direction perpendicular to the patch plane. In some embodiments, a depth patch image may be packed into an image frame with other attribute patch images. For example, a decoder may receive one or more image frames comprising packed patch images as generated by the encoder described above. At least one of the image frames may have been down-scaled at an encoder, and the decoder may up-scale the down-scaled image frame, in a 2D video domain.

The decoder is further configured to video decode the one or more video encoded image frames comprising the patch images. In some embodiments, the decoder may utilize a video decoder in accordance with the High Efficiency Video Coding (HEVC) standard or other suitable standards such as, the Advanced Video Coding (AVC) standard, the AOMedia Video 1 (AV1) video coding format, etc. In some embodiments, the decoder may utilize an image decoder in accordance with a Motion Picture Experts Group (MPEG) or a Joint Photography Experts Group (JPEG) standard, etc.

The decoder is further configured to receive or determine relationship information indicating relationships between the respective attribute patch images, depth patch images, and/or image frames. For example, the decoder may receive relationship information in a compressed point cloud file, wherein the relationship information was determined by an encoder. Also, in some embodiments, the decoder may determine relationship information based on information included in a compressed point cloud file, such as an occupancy map and/or auxiliary information for the compressed point cloud. In some embodiments, the decoder may utilize the received or determined relationship information to adjust one or more parameters used to video decode the video encoded image frames.

The decoder is further configured to determine, for each patch, spatial information for the set of points of the patch based, at least in part, on the attribute patch image comprising the set of points of the patch projected onto the patch plane and the geometry patch image comprising the depth information for the set of points of the patch, and generate a reconstructed version of the compressed point cloud based, at least in part, on the determined spatial information for the plurality of patches and the attribute information included in the patches.

In some embodiments, a method includes receiving one or more encoded image frames comprising patch images for a plurality of patches of a compressed point cloud, wherein, for each patch, the one or more encoded image frames comprise: an attribute patch image comprising a set of points of the patch projected onto a patch plane and a geometry patch image comprising depth information for the set of points of the patch, wherein the depth information indicates depths of the points of the patch in a direction perpendicular to the patch plane. The method further includes receiving an occupancy map for the one or more image frames. The method further includes decoding the one or more encoded image frames comprising the patch images. Additionally, the method included up-scaling one or more of the image frames that were previously down-scaled. The up-scaling is performed in the 2D domain prior to reconstructing a 3D representation of the compressed point cloud. In some embodiments, decoding may be performed in accordance with the High Efficiency Video Coding (HEVC) standard or other suitable standards such as, the Advanced Video Coding (AVC) standard, an AOMedia Video 1 (AV1) video coding format, etc. In some embodiments, decoding may be performed in accordance with a Motion Picture Experts Group (MPEG) or a Joint Photography Experts Group (JPEG) standard, etc.

The method further includes determining, for each patch, spatial information for the set of points of the patch based, at least in part, on the patch image comprising the set of points of the patch projected onto the patch plane and the patch image comprising the depth information for the set of points of the patch, and generating a reconstructed representation of the compressed point cloud based, at least in part, on the determined spatial information for the plurality of patches.

In some embodiments, a non-transitory computer-readable medium stores program instructions that, when executed by one or more processors, cause the one or more processors to implement an encoder as described herein to compress geometry and attribute information of a point cloud.

In some embodiments, a non-transitory computer-readable medium stores program instructions that, when executed by one or more processors, cause the one or more processors to implement a decoder as described herein to decompress geometry and attribute information of a point cloud.

