1. A method for constructing a representation of a planar object comprising: obtaining a depth image of a physical scene as captured from a viewing position, the depth image comprising a plurality of depth values and corresponding to a plurality of points in the physical scene, each depth value indicating a distance from the viewing position to one of the plurality of points in the physical scene; identifying a planar surface along which the planar object is estimated to be positioned; constructing a support map, the support map indicating portions of the planar surface occupied by the planar object and possible surfaces of objects resting on top of the planar object; constructing an occlusion map, the occlusion map indicating portions of the planar surface where the planar object is missing; and constructing a representation of at least one boundary of the planar object, using the occlusion map and the support map.