1. An image processing apparatus comprising: a memory that stores instructions; and a processor configured to execute the instructions to act as: an extraction unit that extracts a color boundary of an image; an influence value calculation unit that calculates an influence value indicating the degree of an influence of the color boundary on a pixel; a setting unit that sets a coefficient for a convolution operation in accordance with the color boundary extracted in the extraction unit and the influence value calculated in the influence value calculation unit; an operation unit that performs a convolution operation using the coefficient set by the setting unit; and a distance calculation unit that calculates a distance from the pixel to the color boundary extracted in the extraction unit, wherein the setting unit sets the coefficient using the distance calculated in the distance calculation unit together with the influence value, the influence value is calculated based on an absolute value of a difference of a pixel color from an average color of a local region or a color difference within a local region, and the influence value becomes larger as the pixel is closer to the color boundary and the influence value becomes smaller as the pixel is farther from the color boundary.