1. A tracked object determination device comprising: a movement state change point detection unit which detects a movement state change point as a time when a movement state determined by a motion vector of a moving object changes; a shooting state change point detection unit which detects a shooting state change point as a time when a shooting state determined by a motion vector of a camera motion of said video changes; a tracking score calculation unit which with a difference between the time of said shooting state change point and the time of said movement state change point as a time difference of said moving object, calculates a tracking score of said moving object based on the time difference of said moving object; and a tracked object determination unit which determines whether said moving object is an object tracked by a cameraman based on the tracking score of said moving object.