1. A paper sheet processing apparatus for performing a recognizing process of a paper sheet, the paper sheet processing apparatus comprising: an image reader that reads an image of a paper sheet; a number-recognition condition memory that stores number recognition conditions for recognizing a unique number given to the paper sheet; and an image processing unit/character recognizing unit that recognizes the unique number printed on the paper sheet from an image read by the image reader, based on the number recognition conditions stored in the number-recognition condition memory, wherein the image processing unit/character recognizing unit comprises: an image processing unit that clips an image of a number portion of the paper sheet from an image of the paper sheet read by the image reader; and a character recognizing unit that recognizes the unique number printed on the paper sheet from the image of the number portion clipped by the image processing unit, and wherein the number recognition conditions stored in the number-recognition condition memory comprise size change information indicating whether to perform a size changing process of the image of the number portion clipped by the image processing unit.