1. A phase measurement system using a phase shift method, comprising: a pattern light projector which projects periodic pattern light on a sample; an image capturing device which captures an image of said pattern light projected on said sample by said pattern light projector; a first reference image memory which stores a first reference image, captured by said image capturing device, of an x direction pattern light having a first pitch and an x direction, and said first reference image memory further stores a second reference image, captured by said image capturing device, of a y direction pattern light having said first pitch and a y direction; a measuring image memory which stores a first measuring image, captured by said image capturing device, of first captured pattern light having said x direction and having a second pitch that is smaller than a value calculated based on the phase resolution of said x direction pattern light having said first pitch and said measuring image memory further stores a second measuring image, captured by said image capturing device, of second captured pattern light having said y direction and said second pitch; a phase data converter which converts said first and second reference images and said first and second measuring images into respective phase data; a first reference phase image data memory which stores a first reference phase image as representing the phase data of said first reference image converted by said phase data converter and said first reference image memory further stores a second reference phase image as representing the phase data of said second reference image converted by said phase data converter; a measuring phase image memory which stores a first measuring phase image as representing the phase data of said first measuring image converted by said phase data converter, and said measuring phase image memory further stores a second measuring phase image as representing the phase data of said second measuring image converted by said phase data converter; and a phase coupler which corrects said first and second measuring phase images based on said first and second reference phase images.