1. An image processing apparatus comprising: a silhouette image generator to generate a three-dimensional (3D) silhouette image of a target object using both an input depth image and an input color image matched with the input depth image; skeleton model generating unit to generate a skeleton model of the target object using the 3D silhouette image; a 3D model generator to generate a 3D model of the target object by matching a template mesh model with the skeleton model of the target object using at least one processor; and a 3D model refinement unit to refine the 3D model of the target object by refining the template mesh model matched with the skeleton model of the target object using the 3D silhouette image.