1. A scene matching reference data generation system comprising: a data input unit to which a set of probe data is input, wherein the set of the probe data includes captured images sequentially obtained by a plurality of probe cars each of which has a function of calculating a vehicle position, and the vehicle positions of the probe cars, as image-capturing positions of the captured images; a temporary storage unit in which the captured images, which are included in the set of the probe data, are temporarily stored; an image similarity degree evaluation unit that evaluates image similarity degrees of the captured images stored in the temporary storage unit, and assigns the image similarity degrees to the captured images; a processing target data selection unit that selects, as a plurality of processing target captured images, a plurality of the captured images with the image similarity degrees equal to or higher than a predetermined degree, from among the captured images stored in the temporary storage unit; a representative image-capturing position determination unit that determines a representative image-capturing position that is a representative of the image-capturing positions of the plurality of the processing target captured images; an image feature point data generation unit that generates image feature point data based on the plurality of the processing target captured images; and a reference data generation unit that generates the reference data used for scene matching to recognize an image of a scene viewed from a vehicle, by associating the image feature point data with the representative image-capturing position.