1. A method for processing keypoint trajectories in a video, wherein the keypoint trajectories describe motion of a plurality of keypoints across pictures of the video over time, comprising steps of: acquiring the video of a scene using a camera; detecting keypoints and associated feature descriptors in each picture; matching the keypoints and associated features descriptors between neighboring pictures to generate keypoint trajectories, each keypoint trajectory specifies locations of a corresponding keypoint in different pictures of the video; coding predictively locations of the keypoints in the keypoint trajectories into a bitstream, wherein the coding includes determining a residual between a current location of a keypoint in a picture and a location of the keypoint predicted using a reference keypoint trajectory, and adding the residual into the bitstream, such that the keypoint trajectories are coded in scalable manner so that all the keypoints of a first trajectory are coded before a current trajectory, and a previous keypoint in a keypoint trajectory is used to predict a later keypoint of the keypoint trajectory, and a keypoint prediction is implemented using a first order predictor; and outputting the bitstream, wherein the steps are performed in a processor.