1. A method of performing joint depth estimation and semantic labeling of an image by one or more computing devices, the method comprising: estimating global semantic and depth layouts of a scene of the image through machine learning by the one or more computing devices; estimating local semantic and depth layouts for respective ones of a plurality of segments of the scene of the image through machine learning by the one or more computing devices, the local depth layouts including relative depth values for an individual pixel in the image representing a depth of the individual pixel in relation to other pixels; and merging the estimated global semantic and depth layouts with the estimated local semantic and depth layouts by the one or more computing devices to semantically label and assign a depth value to the individual pixels in the image.