1. An image processing apparatus comprising: a map generator to generate a color difference map based on a color difference value between a reference channel image and each of remaining channel images among input channel images; an edge direction decision unit to determine an edge direction for each of the pixels constituting the reference channel image; a filtering unit to filter the color difference map after applying an anisotropic filter window to each of the pixels constituting the color difference map, based on the determined edge direction; and an image restoration unit to restore one or more of the remaining channel images by adding the filtered color difference map and the reference channel image.