1. A method performed by one or more computers for rendering an image, comprising: determining a respective final color value for each of a plurality of pixels of the image, comprising: maintaining current color data for each of a plurality of render elements generated by sampling color contribution values for the render element for the pixel, wherein each of the plurality of render elements represents a contribution of a respective type of color calculation to the final color value for the pixel, and wherein the current color data for each of the plurality of render elements comprises a measure of statistical variance of the color contribution values sampled for the render element; determining, for each of the plurality of render elements, whether or not to sample another color contribution value for the render element from the current color data for the render element for the pixel and the current color data for the render element for one or more pixels that neighbor the pixel in the image, including: determining a neighborhood weighted variance from the measure of statistical variance for the render element for the pixel and for the one or more pixels that neighbor the pixel in the image, determining whether or not the neighborhood weighted variance is smaller than a threshold value, and determining whether or not to sample another color contribution value for the render element based on whether or not the neighborhood weighted variance is smaller than the threshold value; and in response to determining to sample another color contribution value for one or more render elements of the plurality of render elements while determining not to sample another color contribution value for one or more other render elements of the plurality of render elements, sampling a new color contribution value only for the one or more render elements for which it was determined to sample another contribution value and refraining from sampling a new color contribution value for the one or more other render elements for which it was determined not to sample another contribution value; and rendering the image for display based on the respective final color values for the plurality of pixels.