1. A method of detecting foreground regions in a panoramic video, the method comprising: receiving, by a computing device, panoramic video data comprising a plurality of panoramic frames and video frame sequence data indicating a video frame sequence for the plurality of panoramic frames, wherein a first panoramic frame of the plurality of panoramic frames comprises a first plurality of pixel values and first spatial data describing a spatial arrangement of the first plurality of pixel values on a two-dimensional grid described by an X-axis and a Y-axis; selecting, by the computing device and from the first panoramic frame, a plurality of scene point locations positioned on the two-dimensional grid, wherein a first scene point location of the plurality of scene point locations depicts a first scene point from a scene shown by the panoramic video depicted by the panoramic video data, and wherein the first scene point location is at a first position on the two-dimensional grid corresponding to a location of a first pixel value of the first plurality of pixel values; identifying, by the computing device, a first prior location depicting the first scene point at a second panoramic frame before the first panoramic frame in the video frame sequence; identifying, by the computing device, a second prior location depicting the first scene point at a third panoramic frame before the second panoramic frame in the panoramic video; determining, by the computing device, a first trajectory for the first scene point, wherein the first trajectory is a vector comprising a concatenation of a first coordinate set for the first scene point location, a second coordinate set for the first prior location and a third coordinate set for the second prior location; determining, by the computing device a plurality of columns from the first panoramic frame, wherein the first scene point location is in a first column of the plurality of columns; randomly selecting, by the computing device, a second scene point location in the first column, a third scene point location in the first column, and a fourth scene point location in the first column; generating, by the computing device, a first vector subspace, wherein a second trajectory of a second scene point depicted at the second scene point location, a third trajectory of a third scene point depicted at the third scene point location, and a fourth trajectory of a fourth scene point depicted at the fourth scene point are basis vectors of the first vector subspace; determining, by the computing device, a first projection error for the first scene point to the first vector subspace; determining, by the computing device, that the first projection error is greater than a projection error threshold; and writing, by the computing device, an indication of the first pixel value to a listing of foreground pixel values.