1. A motion vector detection apparatus for detecting a motion vector for an entire image, comprising: a motion vector detection unit configured to detect motion vectors for respective blocks obtained by dividing an image into the respective blocks each having a predetermined size; a classifying unit configured to classify the respective blocks into groups based on the motion vectors detected by said motion vector detection unit; a representative motion vector calculation unit configured to calculate representative motion vectors which represent the motion vectors of the blocks that belong to the groups classified by said classifying unit; a first determination unit configured to determine whether each of the groups classified by said classifying unit corresponds to a first group or a second group which moves relative to the first group; and a selection unit configured to, when the number of blocks included in the second group exceeds a threshold, select the representative motion vector corresponding to the second group as the motion vector for the entire image, and to, when the number of blocks included in the second group does not exceed the threshold, select the representative motion vector corresponding to the first group as the motion vector for the entire image.