1. A method of coding video, comprising the steps of: constructing motion-compensated extended reference frames from at least portions of first level motion-compensated residual image frames; predicting a second level of motion-compensated residual image frames from the motion-compensated extended reference frames and the video; and fine granular scalable coding the second level motion-compensated residual image frames into a data-stream of temporal frames.