1. A defect detection apparatus for detecting a defect on an imaged object surface having a background pattern from a multi-valued image captured by an imaging device, the defect detection apparatus comprising: a size setting and accepting device for accepting a setting of a size of a defect as a detection object; a size storing device for storing the size of the defect for which the setting has been accepted by the size setting accepting device; a direction instruction accepting device for accepting an instruction in relation to a first direction of the background pattern, wherein the background pattern has a uniformly continuous pattern in at least one direction; a direction storing device for storing the first direction for which the instruction has been accepted by the direction instruction accepting device; an image reducing device for generating a reduced image, the reduced image being reduced from the multi-valued image in the first direction of the background pattern, which is stored in the direction storing device and the reduced image being reduced such that a reduction ratio in the first direction is different from a reduction ratio in a second direction which is perpendicular to the first direction, with an image reduction ratio according to the size of the stored defect; a filter processing device for performing filter processing in the first direction on the reduced image for removing a defect in the reduced image; an image enlarging device for generating a first enlarged image obtained by enlarging the reduced image, subjected to the filter processing by the filter processing device, in the first direction with an image enlargement ratio corresponding to the reciprocal of the image reduction ratio; and a difference calculating device for generating a difference image by calculating a difference between the multi-valued image and the first enlarged image.