1. A method of determining surface normal estimates for a surface of an object which is visible in one or more images of a scene, wherein the object is represented by geometry constructed from the one or more images of the scene, the method comprising: obtaining surface normal estimates for the surface of the object, the surface normal estimates representing a first surface normal field which comprises a non-conservative component and a conservative component; constructing a height map for said surface of the object based on the conservative component of the first surface normal field; using the height map to determine refined surface normal estimates for the surface of the object, the refined surface normal estimates representing a conservative surface normal field for said surface of the object; and storing the refined surface normal estimates for subsequent use in rendering an image of the object.