1. A video encoding method in a temporal direct mode in a hierarchical structure, comprising: calculating a temporal position of at least one of a forward reference frame and a backward reference frame of a high-pass frame; searching for a frame which exists at the temporal position and has motion vector information; if plural frames are searched for, selecting as a co-located frame a frame, among the plural frames, having a motion vector of which an interval is closest to a difference in interval between the forward reference frame and the backward reference frame of the high-pass frame, and extracting the motion vector of the selected frame; and encoding information on the selected frame.