1. An image processing apparatus generating an enlarged image from a binary image with smoothing a neighborhood of a step between two edges, comprising: a step detecting unit configured to detect a first step pixel and a second step pixel in the binary image and step directions of the first step pixel and the second step pixel, the first step pixel forming a step of a first pixel value, the second step pixel forming a step of a second pixel value that is an inverted value of the first pixel value; and an enlargement processing unit configured to invert pixel values of pixels from the first pixel value to the second pixel value in a pixel area corresponding to the first step pixel in the enlarged image, and invert pixel values of pixels from the second pixel value to the first pixel value in a pixel area corresponding to the second step pixel, the pixels of which pixel values are inverted from the first pixel value to the second pixel value being located at positions corresponding to the step direction, the number of the pixels of which pixel values are inverted from the first pixel value to the second pixel value corresponding to the number of pixels that have same pixel values and are continuously arranged in line from the first pixel value in the binary image, the pixels of which pixel values are inverted from the second pixel value to the first pixel value being located at positions corresponding to the step direction, the number of the pixels of which pixel values are inverted from the second pixel value to the first pixel value corresponding to the number of pixels that have same pixel values and are continuously arranged in line from the second pixel value in the binary image.