1. A phase measurement system using a phase shift method, comprising: a pattern light projector which projects periodic pattern light on a sample; an image capturing device which captures an image of said pattern light projected on said sample by said pattern light projector; a first reference image memory which stores a first reference image, captured by said image capturing device, of pattern light having a first pitch; a measuring image memory which stores a measuring image, captured by said image capturing device, of pattern light having a second pitch that is smaller than a value calculated based on the phase resolution of said pattern light having said first pitch; a phase data converter which converts said first reference image and said measuring image into respective phase data; a first reference phase image data memory which stores a first reference phase image as representing the phase data of said first reference image converted by said phase data converter; a measuring phase image memory which stores a measuring phase image as representing the phase data of said measuring image converted by said phase data converter; and a phase coupler which corrects said measuring phase image based on said first reference phase image.