1. A computer-implemented process for generating an interactive viewpoint video, comprising using a computer to perform the following process actions: inputting a plurality of synchronized video streams each depicting a portion of the same scene captured from different viewpoints, wherein said viewpoints form a grid; inputting calibration data defining geometric and photometric parameters associated with each video stream; and for each group of contemporaneous frames from the synchronized video streams, generating a 3D reconstruction of the scene, using the reconstruction to compute a disparity map for each frame in the group of contemporaneous frames, and for each frame in the group of contemporaneous frames, identifying areas of significant depth discontinuities based on its disparity map, generating a main layer comprising pixel information associated with areas in a frame that do not exhibit depth discontinuities exceeding a prescribed threshold and background pixel information from areas having depth discontinuities above the threshold, and a boundary layer comprising foreground pixel information associated with areas having depth discontinuities that exceed the threshold, to produce a layered representation for the frame under consideration.