1. A three-dimensional shape measurement method which applies light emitted from a light source to a sample and a reference surface and which images an interference figure produced from lights reflected from the sample and the reference surface to measure the three-dimensional shape of the sample, the method comprising: adjusting the relative distance between the sample and the reference surface; sequentially imaging the interference figures to acquire color interference images every time the relative distance is changed; separating the acquired color interference image into color components; calculating a maximum amplitude value from an interference waveform for each of the separated color components; and composing the calculated maximum amplitude values to generate color luminance data.