1. An image processing apparatus comprising: a dividing unit constructed to divide an image into a plurality of first image blocks, each of which is a partial area of the image; an acquiring unit constructed to divide each first image block obtained by the dividing means, and to acquire a plurality of compartments that correspond to filters for determining an edge direction of an image of the first image block; a first determining unit constructed to determine the edge direction of each first image block, based on the plurality of compartments acquired by the acquiring unit and the filters; a deciding unit constructed to decide that a first image block whose edge direction cannot be determined by the first determining unit is a failed image block; a generating unit constructed to generate, from the failed image block, a second image block constituted by a plurality of compartments that differ in size from the plurality of compartments acquired by the acquiring unit; a second determining unit constructed to determine the edge direction of the second image block, based on the plurality of compartments generated by the generating unit and the filters; a non-transitory computer-readable memory constructed to store computer-executable process steps; and a processor constructed to function as at least one of the units by executing the computer-executable process steps stored in the memory.