TECHNICAL FIELD

The present disclosure relates to generating and manipulating multi-view interactive digital media representations.

With modern computing platforms and technologies shifting towards mobile and wearable devices that include camera sensors as native acquisition input streams, the desire to record and preserve moments digitally in a different form than more traditional two-dimensional (2D) flat images and videos has become more apparent. Traditional digital media formats typically limit their viewers to a passive experience. For instance, a 2D flat image can be viewed from one angle and is limited to zooming in and out. Accordingly, traditional digital media formats, such as 2D flat images, do not easily lend themselves to reproducing memories and events with high fidelity.

Current predictions (Ref: KPCB “Internet Trends 2012” presentation”) indicate that every several years the quantity of visual data that is being captured digitally online will double. As this quantity of visual data increases, so does the need for much more comprehensive search and indexing mechanisms than ones currently available. Unfortunately, neither 2D images nor 2D videos have been designed for these purposes. Accordingly, improved mechanisms that allow users to view and index visual data, as well as query and quickly receive meaningful results from visual data are desirable.

Overview

Various embodiments of the present invention relate generally to systems and methods for analyzing and manipulating images and video. According to particular embodiments, the spatial relationship between multiple images and video is analyzed together with location information data, for purposes of creating a representation referred to herein as a multi-view interactive digital media representations. The multi-view interactive digital media representations can be output to a device with a display, such as a mobile device, tablet computer or laptop computer.

Multi-view interactive digital media representations can include images of an object from many different viewing angles. Images with viewing angles about a common axis can be grouped together. These images can be provided in a sequence where the viewing angle changes from image to image in the sequence in an orderly manner. Thus, as the sequence of images is viewed on a display, the object can appear to rotate about the common axis. In particular embodiments, a multi-view interactive digital media representation can be provided with images with viewing angles about one or more axes. Thus, when viewed the object in the multi-view interactive digital media representation can appear to rotate about the one or more axes.

In more detail, a multi-view interactive digital media representation can be generated from live images captured from a camera. The live images can include an object. An angular view of the object captured in the live images can be estimated using sensor data from an inertial measurement unit such that an angular view can be associated with each of the live image. The angular view associated with each live image can be used select live images. For example, live images separated by an angular view amount, such as one, two, three, five, ten degrees, etc., can be selected.

The multi-view interactive digital media representation can include a plurality of images where each of the plurality of images includes the object from a different camera view. For example, the live images separated by an angular view amount can be selected. When the plurality of images is output to a display, the object can appear to undergo a 3-D rotation through the determined angular view where the 3-D rotation of the object is generated without a 3-D polygon model of the object.

One method can be generally characterized, as comprising, on a mobile device including a processor, a memory, a camera, an inertial measurement unit, a microphone, a GPS sensor and a touchscreen display, receiving a request to generate a multi-view interactive digital media representation of an object. Then, a sequence live images can be received from the camera on the mobile device where the live images can include 2-D pixel data. The camera can move along a path where an orientation of the camera varies along the path such that the object in the sequence of the live images is captured from a plurality of camera views.

Based upon sensor data from the inertial measurement unit, angular changes in the orientation of the camera along the path can be determined. Based upon the angular changes, an angular view of the object captured in the sequence of the live images can be determined. From the sequence of the live images, the multi-view interactive digital media representation can be generated.

The multi-view interactive digital media representation can include a plurality of images where each of the plurality of images can include the object from a different camera view. When the plurality of images is output to the touchscreen display, the object can appear to undergo a 3-D rotation through the angular view where the 3-D rotation of the object is generated without a 3-D polygon model of the object. A value of the angular view of the object captured in the multi-view interactive digital media representation can output, such as via the touchscreen display.

In another embodiment, a method can be generated on a mobile device. The mobile device can include a processor, a memory, a camera, an inertial measurement unit, a microphone and a touchscreen display. The method can be generally characterized as 1) receiving, via an input interface, on the mobile device, a request to generate a multi-view interactive digital media representation of an object, 2) receiving via the input interface an angle profile where the angle profile can include a plurality of angles or angle information used to determine the plurality of angles, 3) receiving live images from the camera on the mobile device as the mobile device moves along a path where an orientation of the camera varies along the path such that the object in the live images is captured from a plurality of camera views; 4) based upon sensor data from the inertial measurement unit, determining angular changes in the orientation of the camera along the path; 5) based upon the angular changes, determining an angular view of the object captured in each of the live images; 6) based upon the determined angular view of the object in each of the live images and the plurality of angles in the angle profile, selecting a sequence of images from among the live images where one of the live images is selected for each of the plurality of angles; and 7) generating from the sequence of the images the multi-view interactive digital media representation. The multi-view interactive digital media representation can include a plurality of images where each of the plurality of images includes the object from a different camera view. When the plurality of images is output to the touchscreen display the object appears to undergo a 3-D rotation through the angular view wherein the 3-D rotation of the object is generated without a 3-D polygon model of the object.

