BACKGROUND

The state of video has remained largely unchanged for decades. While some attempts have been made to modernize video through the addition of interactivity, these attempts have fallen short for various reasons. For one, the information provided through previous attempts at interactivity has generally been extremely limited. This is largely because the manpower required for manually annotating a video including hundreds of thousands of frames is prohibitively expensive. In addition, previous interactive video interfaces have been unintuitive and very disruptive to the viewing experience.

SUMMARY

Embodiments of the present technology address problems with annotating videos and displaying interactive video interfaces. First, a machine learning engine that can automatically generate metadata for media content reduces the manpower required for thorough annotation of content. Second, new interactive interfaces seamlessly relay this metadata to users without interrupting the users' goal of viewing content. These two elements combined allow for the broad distribution and adoption of detailed interactive video.

The present technology includes a method for identification of objects featured in video content and displaying user interfaces that enables users to retrieve more information about the identified objects. Examples of user interfaces that can be used individually or together with a real-time image recognition engine are discussed with reference to an Overlaid Video Player user interface and an Adjacent Layout user interface. Together, the Overlaid Video Player user interface and the Adjacent Layout user interface address a variety of technical problems associated with current technology for real-time image recognition, real-time retrieval of information of recognized objects and other suitable user-initiated processes related to objects recognized in video content. Although the examples of the subject technology are discussed in the context of user interactions related to browsing and buying items featured in video content, the subject technology could be employed in contexts such as surveillance, evidence gathering, and other suitable contexts.

For instance, current solutions for purchasing products related to video content are difficult to use and have a low purchase rate. To increase engagement and purchase click through rate, the Overlaid Video Player makes purchasing products through video content more intuitive for users via an interface that displays product images in real-time alongside the video. It presents the products on top of the video in a non-intrusive way that allows users to easily click to purchase in a minimal number of steps. The Adjacent Layout can also bring more visibility to the products in video content and are another opportunity for users to make product purchases.

Other technical problems include updating the Overlaid Video Player's interface often enough to provide a seamless display. In order to maintain context with the video, the Overlaid Video Player interface should be updated frequently (e.g., at a frequency of about 1-4 times per second). This rate is too rapid to execute a network request for each interval.

To address this problem, the Overlaid Video Player downloads key data for each video segment (e.g., 0-15 minutes of video content) at the beginning of the segment in a compressed format and caches the key data on the user's device. This allows the interface to be updated frequently by using the cached data on the device, which is much faster than performing an external network request to fetch the data.

Compatibility is another technical consideration for the Overlaid Video Player user interface. The Overlaid Video Player user interface should interface with many different underlying video player technologies (e.g., HTML5, Flash, etc.) in order to be distributed broadly. This includes being able to play/pause/seek the video, and being informed about the state of the video (e.g., current time, playing/paused, whether an ad is playing, etc.). To address this problem, the Overlaid Video Player interface may be coupled to a generic relay (FIG. 1B—Metadata Controller) that eases communication between the Overlaid Video Player interface and the underlying video player. This generic relay makes it easy to plug the interface into new video players.

Embodiments of the present technology include various methods and systems. Some of these methods include receiving, by an electronic device, a request from a user for video content; requesting, by the electronic device, the video content from a video server; and requesting, by the electronic device, metadata about an object associated with a person appearing in the video content from a metadata server. Respective pieces of the metadata are aligned with respective frames of the video content. The electronic device shows, on its display, a first frame of the video content to the user. This first frame includes an image of the person and may or may not show the object associated with the person. In response, the electronic device receives a request from the user for the metadata about the object associated with the person. And in response to the request from the user, the electronic device displays a first piece of metadata about the object with the video content. This first piece of metadata may be aligned with the first frame.

In some examples of these methods, receiving the metadata includes receiving product identification information for objects worn by every person appearing in the video content. This metadata may include links for obtaining the objects worn by every person appearing in the video content. The electronic device may display a corresponding link for obtaining the object, and the corresponding link may dynamically redirect the user to a website for obtaining the object. In some cases, the metadata includes a link for obtaining an object similar to the object worn by the person appearing in the video content. The electronic device may display this type of link as well.

The electronic device may receive the user's request in any of a variety of formats. For instance, a microphone of the electronic device may receive the user's request as a voice command or voice query. Or the electronic device may receive the user's request by detecting, proximate to a play bar area of the display of the electronic device, a cursor movement or touch on a touchscreen.

In some examples of these methods, the electronic device stores the metadata about the object in a memory of the electronic device before displaying the frame of the video content to the user. The electronic device may also request metadata about another object associated with another person appearing in a second frame of the video content while displaying the video content to the user. And the electronic device may pause playback of the video content in response to the request from the user.

Other embodiment methods include receiving, at a server, a first request from a user for metadata about an object appearing in a video on a user device and determining, by the server, that a metadata database does not have the metadata about the object. In response to determining that the metadata database does not have the metadata about the object, the server sends a query for an identity of the object appearing in the video to an object recognition server. The server receives the identity of the object appearing in the video from the objection recognition server and obtains the metadata about the object based on the identity of the object. The server then provides the metadata for display to the user, and the user device displays at least a portion of the metadata with the video. If desired, this metadata is stored in the metadata database and retrieved in response to subsequent requests for the metadata.

In some case, the objection recognition server implements at least one neural network that recognizes the object appearing in the video. The neural network may do this by generating an embedding representing the object appearing in the video and performing a comparison of this embedding to a plurality of embeddings stored in an object database. Respective embeddings of the plurality of embeddings represent respective objects of a plurality of objects. The neural network may determine the identity of the object based on the comparison, e.g., by identifying a closest match from among the plurality of objects as the object. If desired, the neural network may identify, based on the comparison, another object similar to the original object from among the plurality of objects. In these cases, the server may provide metadata associated with this other object to the user. And the user device may display at least a portion of the metadata associated with the other object to the user.

It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the inventive subject matter disclosed herein. It should also be appreciated that terminology explicitly employed herein that also may appear in any disclosure incorporated by reference should be accorded a meaning most consistent with the particular concepts disclosed herein.

