1. An image processing apparatus comprising: a dividing unit configured to divide an image into a plurality of first image blocks, each of which is a partial area of the image; an acquiring unit configured to divide each first image block obtained by the dividing means, and acquire a plurality of compartments that correspond to filters for determining an edge direction of an image of the first image block; a first determining unit configured to determine the edge direction of each first image block, based on the plurality of compartments acquired by the acquiring unit and the filters; a deciding unit configured to decide that a first image block whose edge direction cannot be determined by the first determining unit is a failed image block; a generating unit configured to generate, from the failed image block, a second image block constituted by a plurality of compartments that differ in size from the plurality of compartments acquired by the acquiring unit; and a second determining unit configured to determine the edge direction of the second storage block, based on the plurality of compartments generated by the generating unit and the filters.