1. An apparatus for calibrating a depth image, the apparatus comprising: a three-dimensional (3D) point determiner to determine a 3D point in a camera image and a 3D point in a depth image simultaneously captured with the camera image; a calibration information determiner to determine calibration information for calibrating an error in the depth image captured by a depth sensor and a geometric relationship between the depth sensor and a color camera, using the 3D point in the camera image and the 3D point in the depth image; and a depth image calibrator to calibrate the depth image based on the calibration information and the 3D point in the depth image.