1. An apparatus of decoding image, the apparatus comprising: an entropy decoder which extracts from a bitstream information that indicates an intra prediction mode applied to a current prediction unit to be decoded; a reference pixel determining unit which determines reference pixels from among neighboring pixels adjacent to the current prediction unit and filtered neighboring pixels filtered from the neighboring pixels, based on a size of the current prediction unit and an intra prediction mode of the current prediction unit; an intra prediction performing unit which performs intra prediction in the intra prediction mode on the current prediction unit using the extracted information and the determined reference pixels, wherein the image is hierarchically split from a plurality of maximum coding units, according to information about a maximum size of a coding unit, into coding units of coded depths according to depths, wherein a coding unit of a current depth is one of rectangular data units split from a coding unit of an upper depth, wherein the coding unit is split into coding units of a lower depth, independently from neighboring coding units, and wherein the coding unit of the current depth is split into at least one prediction unit.