1. A computer-implemented method comprising: receiving a color image of a multi-color object; receiving coarse depth information for the multi-color object; determining, based on the color image and the coarse depth information, an illumination model and albedo for the multi-color object, the albedo determined at least in part by estimating chromaticity information for the color image and regularizing the albedo with pairwise weights defined by a Gaussian function applied to the estimated chromaticity information; determining, based on the illumination model and the coarse depth information, refined depth information for the multi-color object; and constructing, based on the albedo and the refined depth information, a relightable three-dimensional (3D) surface of the multi-color object.