1. An image reading apparatus for reading images on both sides of a document, the image reading apparatus comprising: a first reading unit configured to read an image on a front side of the document; a second reading unit configured to read an image on a back side of the document; a first color processing unit configured to perform color processing on the image read by the first reading unit; a second color processing unit configured to perform color processing on the image read by the second reading unit; a first color adjusting unit configured to set a color of the image on the front side of the document color-processed by the first color processing unit as an initial target and adjust a color processing parameter used in the second color processing unit such that a color of the image on the back side of the document color-processed by the second color processing unit becomes the same as the initial target; and a second color adjusting unit configured to set the color of the image on the back side of the document color-processed by the second color processing unit as a second target and adjust, in multiple steps, a color processing parameter used in the first color processing unit and the color processing parameter used in the second color processing unit such that both the color of the image on the front side of the document color-processed by the first color processing unit and the color of the image on the back side of the document color-processed by the second color processing unit using the color processing parameter color-adjusted by the first color adjusting unit become closer stepwise to the second target, wherein if a difference between the color of the image color-processed by the first color processing unit and the color of the image color-processed by the second color processing unit exceeds an allowable range, the initial target is set between the color of the image color-processed by the first color processing unit and the color of the image color-processed by the second color processing unit.