1. A scene matching reference data generation system comprising: a data input unit to which a set of probe data is input, wherein the set of the probe data includes captured images sequentially obtained by a plurality of probe cars each of which has a function of calculating a vehicle position, and the vehicle positions of the probe cars, as image-capturing positions of the captured images; a temporary storage unit in which the captured images included in the set of the probe data are temporarily stored; an accuracy reliability degree evaluation unit that evaluates accuracy reliability degrees of the image-capturing positions of the captured images stored in the temporary storage unit, and assigns the accuracy reliability degrees to the captured images; a processing target data selection unit that selects, as a plurality of processing target captured images, a plurality of the captured images with the accuracy reliability degrees equal to or higher than a first predetermined degree, from among the captured images stored in the temporary storage unit; an image feature point data generation unit that extracts image feature points from the plurality of the processing target captured images, and generate image feature point data based on the image feature points extracted from the plurality of the processing target captured images; and a reference data generation unit that generates the reference data used for scene matching to recognize an image of a scene viewed from a vehicle, by associating the image feature point data with a reference image-capturing position corresponding to the image feature point data.