1. A method tot generating a three-dimensional site model of a site that contains objects comprising: (a) providing at least one of aerial imagery and oblique imagery of the site; (b) processing at least one of the aerial imagery and the oblique imagery to generate a plan view model comprising object outlines, wherein said processing further comprises producing an elevation map using key video images; analyzing the elevation map to identify surface patches; computing local orientation of the surface patches within the elevation map, aggregating the surface patches for the key video images with respect to the local orientation of the surface patches; aggregating elevation information amongst the key video images to produce a three-dimensional structure; processing the three-dimensional structure into a plan view model; and (c) producing from the plan view model a three-dimensional site model comprising blocks that represent objects within the site, wherein the objects within the site include at least one building, the rectilinear outline of the at least one building being determined by: computing an orientation histogram for the initial rectangle representing the building and selecting the angle of orientation with the strongest peak as the dominant orientation of the building; computing a gradient field based on image intensity of the building; integrating the gradient field along and perpendicular to the dominant orientation of the building; fitting straight lines to local maximums of the gradient field along a each direction; detecting knot points at the crossings of the straight lines; dividing the initial rectangle into a plurality of smaller rectangles based on the knot points; and traversing the smaller rectangles to find the rectilinear outline of the building.