1. An apparatus for converting a depth image, the apparatus comprising: a discrete depth image generating unit to generate a discrete depth image by quantizing a depth value of an up-sampled depth image; a high-resolution discrete depth image estimating unit to estimate a high-resolution discrete depth image by optimizing an objective function of the discrete depth image based on a high-resolution color image and an up-sampled depth border; and an image conversion unit to convert the up-sampled depth image to a high-resolution depth image by filtering the up-sampled depth image when a difference between discrete depth values of neighboring pixels in the high-resolution discrete depth image is less than a predetermined threshold value.