1. A multi-viewpoint image display device comprising: a display configured to provide a multi-view; a tracker configured to track a position of a face of a user according to a user shifting position; and a controller configured to shift positions of the multi-view image and to provide a shifted multi-view so that first and second target views of the multi-view are provided as user's left-eye and right-eye images, based on the tracked position of the user's face, wherein the multi-view comprises a plurality of sub-views, such that the first and second target views also comprise a plurality of sub-views, wherein the controller is configured to use a first sub-view of the plurality sub-views that constitute the multi-view to replace at least one of the sub-views comprising the first target view and the remaining target views positioned in the same direction as the first target view, and use a second sub-view of the plurality sub-views to replace at least one of the sub-views comprising the second target view and the remaining target views positioned in the same direction as the second target view.