1. A method of improving depth sensor data in real-time based on scene sensor data, the method comprising: aligning depth images of depth-scene image pairs generated by a depth-scene image sensor pair with corresponding scene images wherein, for each depth-scene image pair, the depth image is warped such that locations of depths in the depth image are aligned with locations of corresponding pixels in the scene image; and improving at least some of the aligned depth images based on image data from one or more of the scene images.