1. An image processing apparatus which processes image data representing an image composed of a plurality of pixels comprising: a plurality of image processing units each of which is disposed so as to correspond to each of partial images obtained by dividing a display image corresponding to the image data and processes data of each of pixels composing the partial image with reference to data of peripheral pixels of the pixel, image data corresponding to the partial image is input to the plurality of image processing units; and an image output unit which outputs display image data using data of pixels on which the image processing has been performed by the image processing units, wherein the plurality of image processing units includes at least a first image processing unit which acquires, when performing the image processing on a first partial image, at least data of pixels in a region adjacent to the first partial image, among pixels composing other partial images adjacent to the first partial image, to use the data for the image processing as the data of the peripheral pixels, and a second image processing unit which performs the image processing on a second partial image different from the first partial image and brokers at least part of data of pixels treated as the peripheral pixels by the first image processing unit from an image processing unit which processes a partial image adjacent to the partial image treated by the first image processing unit to the first image processing unit.