1. An information processing apparatus comprising: a characteristic amount extracting unit configured to extract a first plurality of characteristic amounts, which are information expressing characteristics of a video, from the video; a labeling unit configured to identify a first portion of the first plurality of extracted characteristic amounts as corresponding to a person and a second portion of the first plurality of extracted characteristic amounts as corresponding to a background, and further configured to associate the first portion of extracted characteristic amounts with the person and the second portion with the background; an audio analyzing unit configured to separate voice information of the person from background sound based upon lip movement of the person; a matching degree judging unit configured to judge a degree of matching between the associated first portion of extracted characteristic amounts and second characteristic amounts of at least one other video; a comparing unit configured to compare the first extracted characteristic amounts of one scene in the video from which the first extracted characteristic amounts have been extracted and a second plurality of characteristic amounts of at least one scene in the at least one other video; and a relationship inferring unit configured to infer a relationship between the one scene in the video and the at least one scene in the at least one other video based on a comparison result of the comparing unit.