1. An image processing apparatus comprising: an acquisition unit which acquires, for each of respective viewpoints, capturing condition at the viewpoint, and each frame image captured from the viewpoint in accordance with the capturing condition; a selection unit which selects one of the respective viewpoints as a reference viewpoint by using the capturing conditions acquired by said acquisition unit for the respective viewpoints; a predicting coding unit which performs predicting coding for the frame images acquired by said acquisition unit; an intra-coding unit which performs intra-coding for the frame images acquired by said acquisition unit; and an output unit which outputs, for each of the respective viewpoints for each frame, coding result of the frame image captured from the viewpoint by said predicting coding unit or said intra-coding unit, wherein when performing predicting coding for a frame image captured from the reference viewpoint, said predicting coding unit performs predicting coding by referring to the frame image captured from the reference viewpoint without referring to frame images captured from the viewpoints other than the reference viewpoint.