1. An image processing apparatus comprising: an attribute determination unit adapted to determine attributes including at least character and background for respective pixels of an image; and a filtering unit adapted to weight pixel values in either an M×M (M being a natural number) area including a pixel of interest, using an M×M filter, or an N×N (N being a natural number less than M) area including the pixel of interest, using an N×N filter, in accordance with the attributes determined by said attribute determination unit, and to change a pixel value of the pixel of interest based on the weighted pixel values, wherein edge emphasis processing is performed using the M×M filter by said filtering unit in a case where it is determined by said attribute determination unit that an attribute of the pixel of interest indicates the character, and smoothing processing is performed using the N×N filter by said filtering unit in a case where the it is determined by said attribute determination unit indicate that the attribute of the pixel of interest indicates the background; wherein the smoothing processing is performed using the N×N filter in a case where the attribute of the pixel of interest indicates the background and the attribute of another pixel in the M×M area indicates the character, and the smoothing processing is performed using the M×M filter in a case where the attribute of the pixel of interest indicates background and the attribute of every other pixel in the M×M area does not indicate the character.