1. An image processing apparatus to calculate disparities of a plurality of pixels contained in a frame, the image processing apparatus comprising: a temporal constraint unit to determine, for each of a plurality of pixels in a current frame, whether to enforce a temporal consistency based on at least one of motion information and texture information; and a disparity estimator to estimate disparities of the plurality of pixels by calculating a disparity of each pixel in the current frame where the temporal consistency is determined to be enforced based on the determination result, wherein a disparity of a first pixel in the current frame where the temporal consistency is determined to be enforced, is calculated based on a disparity of a second pixel that is in a previous frame and that corresponds to the first pixel, wherein the temporal constraint unit determines to enforce the temporal consistency to pixels classified as a texture area where a texture is detected and a non-motion area where motion is not detected, and determines not to enforce the temporal consistency to pixels classified as a non-texture area where a texture is not detected or a motion area where motion is detected, among the plurality of pixels.