1. An image decoding apparatus, including: an inverse conversion processing section into which is inputted encoded image data that is encoded for each region of a predetermined number of pixels by processing that includes motion compensation predictive coding processing that encodes a prediction error between a reference image and an image of an encoding object, the inverse conversion processing section being configured to obtain the prediction error or a restored image of the encoding object image by inverse processing of the encoding processing; a motion compensation section configured to obtain a restored image for each of the regions of the predetermined number of pixels by adding the prediction error and the reference image; a data amount reducing section configured to generate a reduced size image of the restored image; a storage section configured to store the restored image and the reduced size image of the restored image; and a reference image acquisition section configured to read out the restored image and the reduced size image of the restored image from the storage section for addition processing of the motion compensation section, wherein, when an image of a processing object of the motion compensation section is encoded by an encoding mode that uses reference pixels that include two times or more number of pixels of the region of the predetermined number of pixels, the reference image acquisition section obtains a reference image by reading out the reduced size image of the restored image from the storage section and obtains the restored image from the reduced size image of the restored image by expanding process, and when an image of a processing object of the motion compensation section is encoded by an encoding mode that uses reference pixels that include less than two times a number of pixels of the region of the predetermined number of pixels, the reference image acquisition section obtains a reference image by reading out the restored image from the storage section.