CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims priority under 35 USC 119 from Japanese Patent Application No. 2019-138287 filed on Jul. 26, 2019, the disclosure of which is incorporated by reference herein.

BACKGROUND

Technical Field

Technology disclosed herein relates to a retrieval device, a training device, a retrieval system, a retrieval program, and a training program.

Related Art

Japanese Patent Application Laid-Open (JP-A) No. 2019-95878 discloses technology to query a driver's driving behavior data and driving behavior data similar to the query is extracted and output.

Further, “Weakly Supervised Video Moment Retrieval from Text Queries” (N. C. Mithun et al., CVPR2019) and “TALL: Temporal Activity Localization via Language Query” (J. Gao et al., ICCV2017) disclose technology using a search text query to retrieve video similar to the query.

When there is a desire to search vehicle behavior data representing temporal vehicle behavior, searching for vehicle behavior data using a search text as a query, similarly to with a general search engine, is more preferable than using vehicle behavior data as a query. Moreover, in addition to vehicle behavior data, it is also preferable to retrieve video data (for example vehicle-view video data) that corresponds to such vehicle behavior data.

However, in the technology of JP-A No. 2019-95878, it is necessary to input driving behavior data corresponding to vehicle behavior data as a query. Moreover, the results output in the technology of JP-A No. 2019-95878 are merely driving behavior data.

By contrast, the technologies of “Weakly Supervised Video Moment Retrieval from Text Queries” (N. C. Mithun et al., CVPR2019) and “TALL: Temporal Activity Localization via Language Query” (J. Gao et al., ICCV2017) retrieve video using search text as a query. However, the technologies of “Weakly Supervised Video Moment Retrieval from Text Queries” (N. C. Mithun et al., CVPR2019) (hereafter referred to as Non-Patent Document 1) and “TALL: Temporal Activity Localization via Language Query” (J. Gao et al., ICCV2017) (hereafter referred to as Non-Patent Document 2) are not capable of retrieving vehicle behavior data.

Thus employing the related technologies does not enable the retrieval of video and vehicle behavior data pairs corresponding to a driving scene described in search text.

SUMMARY

A retrieval device according to a first aspect includes a memory, and a processor coupled to the memory, the processor being configured to: acquire a search text, extract a feature corresponding to the search text by inputting the search text to a text feature extraction model configured to extract features from input sentences, the text feature extraction model being pre-trained so as to reduce a loss represented by a difference between a feature extracted from a sentence and a feature extracted from a correctly matched vehicle-view video, and also being pre-trained so as to reduce a loss represented by a difference between a feature extracted from the sentence and a feature extracted from correctly matched vehicle behavior data representing temporal vehicle behavior, compute a text distance for each of a plurality of combinations stored in the memory, each combination associating a text description, including a plurality of sentences, with a vehicle-view video and with vehicle behavior data representing temporal vehicle behavior, the text distance being represented by a difference between a feature extracted from each sentence of the text description associated with the video and the vehicle behavior data, and the feature corresponding to the search text, and output, as a search result, a prescribed number of pairs of video and vehicle behavior data pairs in sequence from the smallest text distance, in accordance with all text distances.

A training device according to a first aspect includes a memory, and a processor coupled to the memory, the processor being configured, for each of a plurality of training data items associating a text description, including a plurality of sentences, with a vehicle-view video and with vehicle behavior data representing temporal vehicle behavior, to: extract a feature of a sentence of training data by inputting the sentence to a text feature extraction model configured to extract features from input sentences, extract a feature of a video corresponding to the same training data by inputting the video to a video feature extraction model configured to extract features from input video, and compute a first loss represented by a difference between the sentence feature and the video feature; extract a feature of a sentence of the training data by inputting the sentence to the text feature extraction model, extract a feature of vehicle behavior data corresponding to the same training data by inputting the vehicle behavior data to a vehicle behavior feature extraction model configured to extract features from input vehicle behavior data, and compute a second loss represented by a difference between the sentence feature and the vehicle behavior data feature; compute an overall loss function unifying the first loss with the second loss; train the text feature extraction model and the video feature extraction model so as to reduce the overall loss function; train the text feature extraction model and the vehicle behavior feature extraction model so as to reduce the overall loss function; and obtain a pre-trained sentence feature extraction model by causing the training processing to be performed repeatedly, until the overall loss function computed by the unifying section becomes smaller than a prescribed threshold.

