1. An image processing apparatus performing a processing to a processed image where a figure is placed in a background image composed of a main color component having a relatively low lightness and a sub color component having a relatively high lightness, so as to remove the sub color component around the figure, the image processing apparatus comprising: a deciding part to decide whether it is necessary to remove the sub color component based on a density value of the main color component, at each position around the figure; a removing part to remove the sub color component based on a result of a decision in the deciding part; and a reference density value setting part to set a reference density value, wherein the deciding part decides whether it is necessary to remove the sub color component by comparing a density value of the main color component with the reference density value, at each position around the figure; the image processing apparatus further comprising: a comparing part to compare a ratio of a region around the figure where a density value of the main color component is the reference density value or higher, with a reference ratio; and a cancel part to cancel a removal processing by the removing part, based on a result of a comparison in the comparing part.