1. An X-ray analyzer, comprising: a radiation source for irradiating an irradiation point on an inspection sample with a radiation; an X-ray detector for detecting a characteristic X-ray and a scattered X-ray which are radiated from the inspection sample to output, as an X-ray intensity, a signal containing energy information on the characteristic X-ray and the scattered X-ray; an analyzer for analyzing the signal; a movable sample stage for shifting the irradiation point relative to the inspection sample within a mapping area set in advance; an optical microscope system for imaging the mapping area and displaying an image of the mapping area; and an X-ray mapping processing section for discriminating an X-ray intensity corresponding to at least one specific element from an analysis result obtained by the analyzer and determining an intensity contrast in which one of a color and lightness is changed in accordance with the X-ray intensity to perform image display at a position corresponding to the irradiation point in the mapping area, wherein the X-ray mapping processing section includes: reference mapping image storing means for storing a reference mapping image in a measurement area, of a reference sample which is of the same type as the inspection sample and is normal; inspection mapping image storing means for storing a test mapping image of the inspection sample in an area corresponding to the inspection area; difference calculation means for calculating a difference between an X-ray intensity of the reference mapping image and an X-ray intensity of the inspection mapping image at each radiation point for each pixel, assigning each obtained difference result to the same radiation point and generating a different mapping image; and difference mapping image output means for outputting the difference mapping image on a display section.