1. A method of processing a multi-view image, the method comprising: extracting disparity information between an image of a first point of view and an image of a second point of view; estimating a motion between two sequential images of the first point of view and a motion between two sequential images of the second point of view; and generating intermediate image data for each point of view using the estimated motion, wherein the estimating comprises acquiring a motion vector between the two sequential images of the first point of view and a motion vector between the two sequential images of the second point of view, and compensating the motion vector between the two sequential images of the first point of view using the extracted disparity information and compensating the motion vector between the two sequential images of the second point of view using the extracted disparity information, and wherein the motion vector for each point of view is compensated so that the intermediate image data for each point of view have a disparity which corresponds to the extracted disparity information.