FIELD

The present invention relates to methods and apparatus for capturing, streaming and/or playback of content, e.g., content which can be used to simulate an environment and/or ongoing action in an environment.

BACKGROUND

In order to simulate an environment images are sometimes captured in the environment and transmitted to a playback device. The images may then be applied to a model of the environment as a texture, e.g., using information sometimes called a texture map or UV map which indicates how to apply the captured image as a texture to the model of the environment.

The capture and stream of images of an environment can be very data intensive particularly if the images are high resolution images of the type now expected by many viewers. The transmission of a large amount of data associated with such an approach can be problematic given data transmission constraints.

In view of the above it should be appreciated that there is a need for improved methods and/or apparatus for communicating information about an environment, e.g., captured image content, to a playback device.

SUMMARY

In various embodiments a content transmission system captures images of an environment. The images are analyzed to identify objects which move. Such objects are considered foreground objects since they pass in front of other objects which they may obscure. The foreground objects are extracted from a captured image and the areas of the captured image where the foreground object was extracted are patched with content from another image captured when the foreground object was not blocking the view of the area from which the image of the foreground object was extracted. In this way a background image without occlusions can be constructed from multiple sequential images which are captured.

In other embodiments, e.g., some augmented reality embodiments, the background is captured separately, e.g., using a camera on the playback device, and foreground objects captured from images captured at another location are inserted as part of a virtual reality or immersive experience into the environment which supplied the background objects. In such a case the foreground objects may be extracted from images captured at one site, e.g—at a basketball game, and presented in images to a user of a playback device as if they were in an environment where background images were captured, e.g., the location where the playback device is located.

For example in one augmented reality embodiment basketball players and a ball which are treated as foreground objects are captured and extracted from images taken at a basketball game and then combined with images of an different environment such as images captured in a room to give the impression that the basketball players are playing in the living room, e.g., on a coffee table.

While the foreground objects may move and change their location relatively frequently, the background in many situations tends to be relatively stable. Accordingly, it should be appreciated that if efficient methods for communicating and updating information about foreground objects could be achieved considerable efficiencies could be achieved in terms of data transmission and rendering of background images portions since the background often makes up a large portion of an environment.

In some embodiments foreground objects are identified in captured images and cut out to be used as textures. The images of multiple foreground objects are packed into a frame along with information about the location of the object in the original environment so that the relative locations of the objects can be maintained with respect to one another when inserted into another environment for display.

In some embodiments depth information is used to determine the shape of individual foreground objects. A default mesh shape, e.g., a mesh in the shape of a small cylinder, is used in the playback device to represent the shape of an individual object. Based on the determined shape of the object the content delivery system determines how the default shape of the object mesh used for foreground objects should be distorted by the playback device to more accurately represent the foreground object. In addition to mesh distortion information, the distribution system includes with the cropped image of the foreground object information about how portions of the image of the foreground object should be combined with the background over which it is layered. Such information is provided in the form of alpha blending values in some embodiments. The alpha blending values can be used as a mask by controlling which portions of the transmitted foreground image will block out the background portion over which the foreground object is layered. Thus while a rectangular image may be communicated as a foreground texture corner, other portions which do not include image content showing the foreground image will not contribute to the final output since an alpha blending coefficient of 0 may be associated with such portions so they do not contribute to the output.

Images of multiple foreground objects to be used as textures are packed into a single frame. Foreground objects corresponding to multiple different playback frames may be, and sometimes are, included in a single transmitted encoded frame with the communicated information indicating which time period or frames a particular foreground object texture relates to.

By communicating textures in the form of cropped images an entire captured image frame need not be communicated to the playback device each time there is a change, e.g., movement, in the captured environment. Furthermore the rate at which foreground object information is updated and communicated to the playback device can different, e.g., faster, than the rate at which background image content is updated.

Numerous variations on the above described methods and apparatus are possible.

