1. An image processing apparatus, comprising: a modeling processor to generate a three-dimensional (3D) model corresponding to an actual object using an input color image and an input depth image corresponding to the input color image; an illumination map calculating processor to generate an illumination map of the 3D model corresponding to the actual object by performing rendering based on the input light environment information with assuming the input depth image as a constant material; and a material map calculating processor to generate a material map of the 3D model by performing a pixel-by-pixel divide operation with respect to the illumination map in the input color image and distinguish a color of the actual object from a background color; a calculating processor to perform photon-based rendering with respect to an input virtual object and the 3D model based on the input light environment information, to generate an irradiance map comprising illumination difference occurring by inserting the virtual object into the 3D model, and to generate a difference image comprising color distortion information in RGB color values of respective pixels occurring by inserting the virtual object into the 3D model using the irradiance map; and a rendering processor to generate a result image comprising the virtual object by synthesizing the difference image and the input color image.