1. A method of coding video, comprising the steps of: coding the video into a data-stream of base layer frames; computing residual image frames in a transform domain from the base layer frames; constructing extended base layer reference frames from the base layer frames and at least portions of the residual image frames; motion-compensating the extended base layer reference frames to produce motion-compensated extended base layer reference frames; predicting motion-compensated residual image frames from the motion-compensated extended base layer reference frames and the video; and fine granular scalable coding the motion-compensated residual image frames into a data-stream of temporal frames.