1. A method for constructing a representation of a planar object comprising: obtaining a depth image of a physical scene as captured from a viewing position, the depth image comprising a plurality of depth values and corresponding to a plurality of points in the physical scene, each depth value indicating a distance from the viewing position to one of the plurality of points in the physical scene; identifying a planar surface along which the planar object is estimated to be positioned; constructing a support map, the support map indicating portions of the planar surface occupied by the planar object and possible surfaces of objects resting on top of the planar object; constructing an occlusion map, the occlusion map indicating portions of the planar surface where the planar object is missing; testing a boundary of the planar surface using the occlusion map and the support map; constructing a boundary of the planar surface based on the testing; constructing a representation of at least one boundary of the planar object, using the occlusion map and the support map; constructing an occlusion gradient map based on the occlusion map and the at least one boundary of the planar object; constructing a support gradient map based on the support map and the at least one boundary of the planar object; and classifying a segment associated with a boundary of the planar object based on the occlusion gradient map and the support gradient map.