1. An object tracking apparatus, comprising; an image input device inputting image sequence data; a memory storing a first image and a second image as the image sequence data; a tracking point indication unit configured to initially indicate a plurality of tracking points on the first image stored in said memory; a pattern matching unit configured to compare a pattern of each tracking point of the first image with the second image stored in said memory, and to detect a plurality of destination candidate points from the second image, wherein each of the destination candidate points corresponding to each of the tracking points; and a position inconsistency correction unit configured to decide whether a positional relation of the plurality of destination candidate points is consistent with a positional relation of the plurality of tracking points, and to correct at least one of the plurality of destination candidate points on the second image if the positional relation of the plurality of destination candidate points is inconsistent with the positional relation of the plurality of tracking points.