1. An image processing apparatus that detects a background color of a document based on image data of a colormetric system in which a brightness component and a color component are separated, comprising: a first element counting unit that divides a color plane defining the color component into a plurality of color areas, and counts a number of elements of the image data belonging to each of the plurality of color areas; a single area evaluation unit that extracts a color area in which the number of elements of the image data exceeds a first threshold value, and gives a positive evaluation to the extracted color area; a second element counting unit that counts a number of elements of the image data belonging to each area of a multiple area constituted of mutually adjacent color areas of the plurality of color areas; a multiple area evaluation unit that extracts a multiple area in which the number of elements of the image data exceeds a second threshold value, and gives a positive evaluation to each color area included in the extracted multiple area; and a color area specifying unit that specifies a color area to which a positive evaluation is given by both the single area evaluation unit and the multiple area evaluation unit as a color area of background color.