1. An image processing device for processing a character rectangle extracted from an image, comprising: an acquiring unit configured to acquire a pixel block, which lies within the character rectangle in contact with an enclosing border of the character rectangle by scanning along the enclosing border, wherein a pixel block not in contact with the enclosing border is not acquired; a determination unit configured to determine whether or not the acquired pixel block has a likelihood of noise; a generating unit configured to generate a noise candidate removed character rectangle by removing from the character rectangle the pixel block when it is determined that the pixel block has the likelihood of noise; and an outputting unit configured to assess validities of both the noise candidate removed character rectangle and the character rectangle by performing character recognition, compare the two validities, and output a character recognition result based on the comparison result, wherein said outputting unit obtains degrees of recognition similarity of the character recognition performed for both the noise candidate removed character rectangle and the character rectangle, assesses the validities based on (i) the degrees of recognition similarity, (ii) a character width of the noise candidate removed character rectangle, (iii) a character height of the noise candidate removed character rectangle, (iv) a character width of the character rectangle, and (v) a character height of the character rectangle, and outputs the character recognition result for one the noise candidate removed character rectangle and the character rectangle with a greater validity.