1. A system for stabilizing an image sequence, comprising using a computer to perform the following process actions: receiving an input image sequence comprising two or more sequential image frames of a scene; evaluating the frames of the input image sequence and automatically learning a probabilistic number of objects to be identified within each frame of the input image sequence and a probabilistic number of layers for each frame; probabilistically decomposing the input image sequence into the learned number of objects and layers by probabilistically inferring transformations for each object and layer from one frame to the next; constructing a stabilized output image sequence by using one or more of the decomposed objects and layers from two or more frames of the input image sequence to create a composite sequence of stabilized image frames; and wherein corresponding objects and corresponding layers are transformed to construct a sequential stabilized alignment of objects and layers between each frame of the stabilized output image sequence by applying an inverse transformation and alignment to each corresponding object and layer for stabilizing each object and layer within the composite sequence of stabilized image frames.