COPYRIGHT NOTICE AND AUTHORIZATION

Portions of the documentation in this patent document contain material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office file or records, but otherwise reserves all copyright rights whatsoever.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a system for automatically segmenting and classifying video content objects within a video, auctioning rights to hyperlinks that associate relevant content with the video objects, and creating an overlay with select video objects which enables a video viewer to interact with video object in the video and interact with the relevant content.

2. Background

Video is the technology of electronically capturing, recording, processing, storing, transmitting, and reconstructing a sequence of still images representing scenes in motion. Video technology was first developed for television systems, but has been further developed in many formats to allow for viewer video recording. Motion pictures on film can be converted into video formats. Video can also be viewed through the Internet (World Wide Web) as downloaded video files or streaming files on computer monitors or mobile phones. Video can also be viewed on social media applications.

Animation is the rapid display of a sequence of images of artwork or model positions in order to create an illusion of movement. It is an optical illusion of motion due to the phenomenon of persistence of vision, and can be created and demonstrated in a number of ways. The most common method of presenting animation is as a motion picture or video, although several other forms of presenting animation also exist.

Video content segmentation is the systematic decomposition of a motion picture into its individual frames in chronological order, and then deconstructing the frame into its objects (components) such as a person, a shirt, a tree, a leave etc. Segmenting video content results in a large number of objects with little value if not classified.

Classification is the process of assigning an object of one frame to the same class of the same object of another frame. It enables the automated recognition that a specific red shirt in one frame is the same as the red shirt in another frame. There are several approaches to assigning video objects to the class they belong to such as by the contours of its appearances in successive video frames. For example, this may be done by matching curvature features of the video object contour to a database containing preprocessed views of prototypical objects. See, “MOCA Project: Object Recognition,” Copyright © 2009 Praktische Informatik IV, University of Mannheim.

For each two-dimensional appearance of an object in a video frame curvature features of its contour are calculated. These features are matched to those of views of prototypical video objects stored in a database. By applying context rules such as “a house may have a car in the frame or may have a tree in the frame but does not have a TV in the frame” the accuracy can be increased. The final classification of the object is achieved by integrating the matching results for successive frames.

There are several paradigms and algorithms for video segmentation and classification. Most are based on segmenting video into layers such as a static background layer and a dynamic foreground layer and using multiple cues, such as spatial location, color, motion, contours and depth discontinuities, etc.

Rotoscoping is an animation technique in which animators trace over live action film movement, frame by frame, for use in animated films. Digital Rotoscoping as disclosed in U.S. Pat. No. 6,393,134 (Tostevin et al.) uses algorithms to create vector outlines of video objects.

By shooting video from several perspectives with synchronized cameras, video segmentation algorithms can be used to automatically reconstruct 3D wireframes of moving objects. If video segments show multiple perspective of one video object, 3D wire frames can be constructed even if only one camera was used as long as the camera captured many perspectives of the object.

In one embodiment of the invention automatic rotoscoping techniques are applied to videos which have been shot by multiple camera angles to reconstruct the 3D objects and save their wireframes in form of vector data into the video object database. In another embodiment rotoscoping techniques are applied to videos which have been shot by one camera but many perspectives of the video objects are available. When a viewer selects an object for which there is 3D information available, the viewer is presented with a means to control the animation of the 3D object such as rotate, move, scale etc. In yet another embodiment of the invention animated 3D objects are positioned into videos to replace or superimpose video objects. These animated 3D video objects may have been derived from digital rotoscoping or may be animated objects from scratch.

An object of the invention is to provide an automated system for segmenting raw video to create an inventory of video objects which may be used to make the video interactive, and to auction these video objects to retailers, brands or affiliates. The term video object is defined as data and visualizations of data which relate to content objects of videos such as a shirt. This data may include image data, or vector graphic data which has been linked to the content object of a video or has been generated from video data.

The invention is not tied to any specific method of segmenting or classifying video content objects.

In one embodiment of the invention an object information library which contains descriptive information and/or meta-data regarding objects which may appear in the video is used to associate meta data such as product information, unique product identification information or stock keeping units with the segmented video objects.

A further object of the invention is to create an electronic market exchange whereby rights to one or an inventory of hyperlinks that are embedded within the interactive layer of video objects and control where a viewer is redirected to if they interact with the video object or objects, are automatically auctioned to a third party such as an retailer, brand, affiliate and/or advertiser.

SUMMARY OF THE INVENTION

A system is provided for identifying objects in video either automatically by using an object database or manually by using information provided by the creator of the video, making the objects interactive in every frame of the video that they appear, allowing the creator to embed information behind the video objects that appears when a viewer hovers over, clicks or touches the object, allowing the creator to describe at least one object that appears in a video, associating a hyperlink with a clickable space around the object in every frame that the object appears in, and electronically auctioning the rights to the hyperlink and what data the hyperlink connects to. The system includes a video server communicating with a consumer server over a distributed network. The video server includes at least one video file, an object inventory database, and relevant content including associated hyperlinks in a database stored on or accessible to the video server. The object inventory database contains information describing a location of at least one video object within at least one video file and the hyperlink that is associated with the object. A video overlay links the relevant content with a given video object and creating a selectable hyperlink whose position tracks a dynamic location of the video object in the video. The consumer server includes a software application for accessing and displaying at least one video file stored in the video database, the software application including means for interacting with the video overlay to select an object displayed in the video file.

More specifically, a system is disclosed for viewing video having segmented video objects. The system includes a video server including a computer or mobile device connectable to a distributed network and having a processor, random access memory, read-only memory, and mass storage memory. At least one video file is stored in a video database on or accessible to the video server. An object inventory database is stored on or accessible to the video server, the object inventory database containing information describing a location of at least one video object within the at least one video file. Object description content is stored in a database stored on or accessible to the video server. A video overlay links the object description content with a given video object and creating a selectable hyperlink whose position tracks a dynamic location of the video object in the video, wherein the video overlay is adapted to allow a consumer to interact with the video overlay to select an object displayed in the video file and be displayed content related to the object that has been added by the creator of the video, and/or content that has been added automatically by the system, or content that has been added a viewer or viewers of the video. The system also allows a viewer to click a button that has a hyperlink associated with it that takes the viewer to an online store where a product or products associated with the object in the video can be purchased. The viewer also has the ability to add the selected object to a collection of objects. The viewer also has the ability to share a hyperlink directly associated with the video and the object with other people over a social network, email, the internet, mobile messaging and any other commonly used electronic forms of communication.

In some embodiments, the video server of the aforementioned system may include a software application for displaying content associated with the object.

In some embodiments, the video server of the aforementioned system may include a software application for storing one of the object or a data associated with the object in a collection of objects. The video server may include software for sharing the collection of objects or sharing a data associated with the collection of objects. The video server may share the collection of objects or data associated with the collection of objects with a social networking service (or social networking server) via the distributed network. The video server may post the collection of objects or data associated with the collection of objects to a social networking server accessible service via the distributed network. The social networking server may issue an alert to at least one user-specified person that data is posted to the server.

According to one aspect of the aforementioned system, the video server includes a software application for searching at least one of the object inventory database and the at least one video file for occurrences of the selected object, and selectively displaying segments of the at least one video file containing the selected object.

According to one aspect of the aforementioned system, the video server includes a software application for searching at least one of the object inventory database and video files in the video database for occurrences of the selected object, and selectively displaying segments of the at least one video file containing the selected object.

According to one aspect of the aforementioned system, the video server includes a software application for purchasing an item associated with the selected object.

According to one aspect of the aforementioned system, the video server includes video consumer software application including a content display application that interacts with the video overlay and displays relevant content when a given video object is selected hovered-over or pointed at with a pointing device.

According to one aspect of the aforementioned system, the video server includes a software application for storing one of the object or a hyperlink associated with the object in at least one user-defined collection of objects.

Also disclosed is a method for trading and/or purchasing rights to hyperlinks that are associated with video objects on an electronic market exchange, comprising providing at least one video file stored in a video database on or accessible to a video server via a distributed network; providing an object inventory database stored on or accessible to the video server via the distributed network, the object inventory database containing information describing a location of at least one video object within the at least one video file; providing a video overlay linking with a given video object and creating a selectable hyperlink whose position tracks a dynamic location of the video object in the video; accessing and displaying the at least one video file; and interacting with the video overlay to select an object displayed in the video file.

The aforementioned method may comprise a step of displaying content associated with the selected object.

The aforementioned method may comprise a step of storing one of the selected object or a hyperlink associated with the selected object in a collection of objects. Moreover, the method may include a step of sharing the collection of objects or sharing a hyperlink associated with the collection of objects via the distributed network. The video server may post the collection of objects or data associated with the collection of objects to a social networking server accessible service via the distributed network.

The aforementioned method may comprise a step of a step of searching at least one of the object information library and the at least one video file for occurrences of the selected object, and selectively displaying segments of the at least one video file containing the selected object.

The aforementioned method may comprise a step of a step of searching at least one of the object information library and video files in the video database for occurrences of the selected object, and selectively displaying segments of the at least one video file containing the selected object.

The aforementioned method may comprise a step of purchasing an item associated with the selected object. The purchasing step may be performed by a software application executed on the video server.

Also disclosed is a method for ranking the duration of object appearances in video files according to a duration of time, frequency and context a given object appears in the video. The method comprises providing a database of video files; searching each video file for the appearances of at least one object; determining a duration of time, frequency and context that the at least one object was displayed in each video file; and for each video file outputting information specifying the title of the video file and duration of time, frequency and context.

The method further includes outputting information specifying a ranking of the duration of time, frequency and context in a given video file relative to each of the other video files in the database.

The method may further include searching each video file for the appearance of first and second objects; determining a duration of time that each of the first object and the second object were displayed in each video file; and for each video file outputting information specifying the title of the video file and each duration of time.

The method includes automatic rotoscoping to reconstruct the 3D objects and save their wireframes in form of vector data into the video object database and allowing users to interact with the 3D objects.

The method includes method of positioning animated 3D objects into videos to replace or superimposed existing video objects.

The method includes documenting user interaction with video objects, searches for video objects and sharing of objects and collections with a social network and providing analytics of such documentation to retailers, brands, affiliates, and/or advertisers. Disclosed is a system for creating an interactive overlay for a video, including a server including a computer having a processor, random access memory, read-only memory, and mass storage memory. The server includes one or more video files stored in a video database; an object inventory database stored on the server and containing information describing a location of at least one video object within one of the video files; and an overlay generation application (software or firmware) executed on the server. The overlay generation application creating a video overlay including at least one selectable hyperlink whose shape and position (coordinates) track a dynamic location of the video object in the video, and storing the overlay on the server.

The method also includes a way to search the video database and the object database by analytics related to the content, such as number of views, number of clicks, number of objects clicked, number of clicks on an object, number of shared videos, number of shared objects, total views for other videos created by the same creator, conversion rates, purchases associated with the content, click through rates, and other analytical data commonly used in online vide and social video.

The method also includes a way to display the analytical information associated with the video including charts and other graphical ways of displaying data.

The system of claim may further include a wireframe database stored on the server, the wire frame database storing a three-dimensional wireframe representation of at least one video object. The aforementioned overlay generation application matches occurrences of video objects in the object inventory library with three-dimensional wireframe representations of at least one video object and creates a video overlay including matched wire frame representations of the at least one video object wherein the shape and position of the wire frame representation tracks a dynamic location of the video object in the video.

Also disclosed is a system for making an interactive overlay for a video, including: a server including a computer having a processor, random access memory, read-only memory, and mass storage memory, the server including one or more video files stored in a video database. The system further includes an object inventory database stored on the server and containing information describing a location of at least one video object within one of the video files; a wire frame database stored on the server, the wireframe database storing a three-dimensional wireframe representations of at least one video object; and an overlay generation application (software or firmware) executed on the server, the overlay generation application matching occurrences of video objects in the object inventory library with three-dimensional wireframe representations of at least one video object and creating a video overlay including matched wire frame representations of the at least one video object wherein the shape and position of the wireframe representation tracks a dynamic location of the video object in the video, and storing the overlay on the server. The video overlay may optionally include at least one selectable hyperlink whose shape and position tracks a dynamic location of the video object in the video.

Also disclosed is a system for making an interactive overlay for a video, including a server including a computer having a processor, random access memory, read-only memory, and mass storage memory, the server including one or more video files stored in a video database. The system further includes an object inventory database stored on the server and containing information describing coordinates of at least one video object within one of the video files; a wireframe database stored on the server, the wireframe database storing three-dimensional wireframe representations of at least one video object; and a wireframe generation application executed on the server, the wireframe generation application selecting a video object from the video object inventory database and generating a three-dimensional wireframe of the selected video object from multiple perspective occurrences of the video object in the video file and storing the three-dimensional wireframe in the wire frame database. The wireframe generation application may optionally generate a lighting model for each occurrence of a video object in a video frame from light distribution on the video object.

The aforementioned system may further include an overlay generation application executed on the server, the overlay generation application matching occurrences of video objects in the object inventory library with three-dimensional wireframe representations of at least one video object and creating a video overlay including matched wire frame representations of the at least one video object wherein the shape and position of the wire frame representation tracks a dynamic location of the video object in the video, and storing the overlay on the server. The video overlay may optionally include at least one selectable hyperlink whose shape and position tracks a dynamic location of the video object in the video.

Also disclosed is a system for automatically segmenting and classifying video content into objects including: a server including a computer connectable to a distributed network and having a processor, random access memory, read-only memory, and mass storage memory, the video segmenting and classification server including one or more video files stored in a video database. The system further includes a wire frame database stored on the server, the wire frame database storing a three-dimensional wireframe representations of at least one video object and at least one of descriptive information and semantic information used to describe the object; an object inventory database containing information describing a location of at least one video object within one of the video files; and a video content analysis application executed on the processor, the video content analysis application segmenting the video files to identify locations of video objects, for each video object searching for a matching occurrence in the wireframe database, retrieving information describing the video object from the wireframe database, and storing in the object inventory database information describing the dynamic location of the video object within the video and information describing the video object.

