1. A disparity estimation apparatus comprising: an energy calculator to calculate an energy related to stereo matching of each of a left image and a right image which constitute a stereo image; a map generator to generate a visibility map for determining an error in disparity between the left image and the right image using the calculated energy; an energy recalculator to recalculate the energy with respect to a region in the visibility map including a visibility error generated due to the error in disparity and to maintain the energy with respect to the visibility map outside the region in the visibility map including the visibility error; and a disparity determiner to determine disparity from the stereo image using the recalculated energy of each of the left image and the right image.