1. A tracking apparatus comprising: a tracking subject setting unit which sets a tracking subject in first image data obtained by imaging; a tracking subject search unit which searches for the set tracking subject from second image data obtained by imaging after the first image data is obtained, by using at least one of luminance of the first image data, color of the first image data, a result of face detection in the first image data, and a result of focus detection of the first image data; an obstructing detection unit which detects, from the second image data, whether the tracking subject is obstructed; and an obstructing area detection unit which detects, from the second image data, an obstructing area which is obstructing the tracking subject when the obstructing of the tracking subject is detected, wherein the obstructing detection unit detects that the tracking subject is obstructed by comparing a standard focus detection information which is a standard information of the tracking subject with a focus detection information obtained at a tracking position, wherein the tracking subject search unit exclusively sets a search area around the obstructing area in the second image data and searches the set search area for the tracking subject in the second image data by using at least one of the luminance of the first image data, the color of the first image data, or the result of face detection in the first image data when the obstructing of the tracking subject is detected in the second image data, wherein the luminance, color, and result of face detection in the first image data are information different from the result of focus detection of the first image data; and wherein the tracking subject search unit searches for the set tracking subject in the second image data by using only the result obtained from focus detection of the first image data, when the obstructing of the tracking subject is not detected in the second image data.