1. A vehicle position detection system comprising: a reference data database in which reference data is stored, wherein the reference data is generated by associating image feature point data generated by extracting image feature points from a captured image obtained by capturing a scene viewed from a vehicle, with an image-capturing position at which an image is captured to obtain the captured image corresponding to the image feature point data, and a recorded vehicle event that indicates a vehicle event at when the image is captured to obtain the captured image; a captured image processing unit to which an actually-captured image is input, and which outputs data for matching generated by extracting image feature points from the actually-captured image, wherein the actually-captured image is obtained by capturing an image of a scene using a vehicle-mounted camera; a vehicle event evaluation unit that generates information on an actual vehicle event that is a vehicle event detected by the vehicle; a first reference data output unit that extracts first reference data whose image-capturing position is located in a vicinity of an estimated position of the vehicle, from among a set of the reference data stored in the reference data database, based on the estimated position of the vehicle, and outputs the extracted first reference data as a matching candidate for the data for matching; a second reference data output unit that extracts second reference data that includes the recorded vehicle event that matches the actual vehicle event, from among the set of the reference data stored in the reference data database, based on the vehicle event, and outputs the extracted second reference data as the matching candidate for the data for matching; a matching performing unit that performs matching between at least one of the first reference data and the second reference data, and the data for matching; and a vehicle position determination unit that determines a position of the vehicle, based on the image-capturing position associated with the reference data that matches the data for matching.