1. A method of classifying a scene for each person in a video, the method comprising: detecting a face within input video frames, using a computer; segmenting the input video frames into a plurality of shots, wherein each shot comprises a segment of input video frames; detecting a shot change of the input video frames from the segmented input video frames; extracting a person representation frame from a shot; performing a person clustering of the extracted person representation frames from the shot using information corresponding to the shot within a time window; detecting a scene change, after a person portion is separated from the input video frames corresponding to the plurality of shots, by extracting background information from a background portion from the input video frames corresponding to the plurality of shots, and by comparing the extracted background information from an input video frame with at least one other input video frame; merging similar clusters from the clustered persons from the extracted person representation frames from the plurality of shots; and connecting scenes including a common person from the merged similar clusters for each person in a plurality of scenes, wherein the performing of the person clustering comprises: receiving current cluster information, a current person representation frame, and a comparison person representation frame to be compared; comparing the current person representation frame and the comparison person representation frame to be compared; determining whether the current person representation frame is similar to the comparison person representation frame to be compared, and as a result of the comparing; extending the time window when the current person representation frame is similar to the comparison person representation frame to be compared; including the comparison person representation frame which has been compared to the current person representation frame in the current cluster information; and setting a subsequent person representation frame as the new comparison person representation frame to be compared in the time window.