1. A banknote processing apparatus that performs an unfit note determination to determine degree of soiling of a banknote being transported, and sorts the banknote in accordance with a determination result, the banknote processing apparatus comprising: a banknote receiving unit that receives the banknote; a transport unit that transports the banknote received by the banknote receiving unit to one of plural output destination units in accordance with a result of the unfit note determination; an image acquiring unit that irradiates the banknote being transported by the transport unit with light of a predetermined wavelength, and acquires an image of the banknote; and an unfit note determining unit that performs the unfit note determination based on the image acquired by the image acquiring unit wherein an image type, a determination area and a coefficient to be used for the unfit note determination are previously set for each banknote orientation of each banknote type, and the unfit note determining unit identifies, based on the banknote type and the banknote orientation of the banknote, the image type to be used for the unfit note determination, the determination are to be used for the image type, and the coefficient for the determination area, calculates an intermediate determination value from the image of an identified image type by applying an identified coefficient to a value calculated from pixel values of pixels in an identified determination area, and determines the degree of soiling of the banknote by using the intermediate determination value.