CROSS-REFERENCE TO RELATED APPLICATION

Priority is claimed on Japanese Patent Application No. 2018-161909, filed Aug. 30, 2018, the content of which is incorporated herein by reference.

BACKGROUND

Field of the Invention

The present invention relates to a learning device, a learning method, and a storage medium.

Description of Related Art

In recent years, in a field of automated driving in which a vehicle autonomously travels, an attempt is made to determine a traveling plan by artificial intelligence (AI) and machine learning. Research and commercialization are also in progress for reinforcement learning that determines an action on the basis of an observation value, calculates a reward based on feedback from an actual environment or a simulator, and optimizes a model parameter. Related to this, a method of generating an operation amount for changing a state of a vehicle by reinforcement learning is disclosed (Japanese Unexamined Patent Application, First Publication No. 2007-65929).

SUMMARY

In a case in which reinforcement learning is applied to automated driving, it is assumed that an application scene spreads not only to an operation amount of a vehicle but also to various technical elements, but in the related art, an application destination of the reinforcement learning is limited. Therefore, there has been a case in which flexible learning is not able to be performed for an action plan part of an automated driving vehicle.

The present invention has been made in consideration of such circumstances, and an object of the present invention is to provide a learning device, a learning method, and a storage medium capable of performing more flexible learning.

The learning device, the learning method, or the storage medium according to the present invention adopts the following constitution.

(1): A learning device according to an aspect of the present invention includes a planner configured to generate information indicating an action of a vehicle, and a reward deriver configured to derive a plurality of individual rewards obtained by evaluating each of a plurality of pieces of information to be evaluated, which include feedback information obtained from a simulator or an actual environment by inputting information based on the information indicating the action of the vehicle to the simulator or the actual environment, and derive a reward for the action of the vehicle on the basis of the plurality of individual rewards. The planner performs reinforcement learning that optimizes the reward derived by the reward deriver.

(2): In the aspect of (1) described above, the reward deriver derives the individual reward by applying each of rules of which evaluation characteristics are different from each other to at least a part of the plurality of pieces of information to be evaluated.

(3): In the aspect of (2) described above, the reward deriver derives the individual reward by applying any of a plurality of reward functions of which distribution shapes of the individual rewards for a relation to a target value are different from each other to at least a part of the plurality of pieces of information to be evaluated.

(4): In the aspect of (1) described above, the reward deriver calculates the reward for the action of the vehicle by multiplying the plurality of individual rewards with each other.

(5): In the aspect of (1) described above, the feedback information includes at least a part of a speed, an acceleration, and a lateral position of the vehicle.

(6): In the aspect of (1) described above, the information to be evaluated includes a risk derived on the basis of the action of the vehicle.

(7): In the aspect of (3) described above, the plurality of reward functions include a reward function that returns a predetermined value in a case in which an input value matches a target value and returns a smaller value as an absolute value of a difference between the input value and the target value increases.

(8): In the aspect of (3) described above, the plurality of reward functions include a reward function that returns a predetermined value in a case in which an input value matches a target value and returns a smaller value as an absolute value of a difference between the input value and the target value increases, however, a degree to which the individual reward for the difference between the input value on a side where the input value exceeds the target value and the target value is reduced is greater than a degree to which the individual reward for a difference between the input value on a side where the input value is less than the target value and the target value is reduced.

(9): In the aspect of (3) described above, the plurality of reward functions include a reward function that returns a predetermined value in a case in which an input value is equal to or greater than a target value and returns a smaller value as an absolute value of a difference between the input value and the target value increases in a case in which the input value is less than the target value.

(10): In the aspect of (3) described above, the plurality of reward functions include a reward function that returns a predetermined value in a case in which an input value is equal to or less than a target value and returns a smaller value as an absolute value of a difference between the input value and the target value increases in a case in which the input value is greater than the target value.

(11): In the aspect of (3) described above, the plurality of reward functions include a reward function that returns an example of a predetermined value in a case in which the input value is within a target range and returns a smaller value as an absolute value of a difference between the input value and an upper limit or a lower limit of the target range increases.

(12): In the aspect of (3) described above, the plurality of reward functions include a reward function that returns a larger value as an input value approaches any of two or more target values.

(13): In the aspect of (12) described above, the reward function returning the larger value as the input value approaches any of the two or more target values returns different individual rewards according to which of the two or more target values the input value matches.

(14): A learning method according to another aspect of the present invention causes a computer to generate information indicating an action of a vehicle, derive a plurality of individual rewards obtained by evaluating each of a plurality of pieces of information to be evaluated, which include feedback information obtained from a simulator or an actual environment by inputting information based on the information indicating the action of the vehicle to the simulator or the actual environment, derive a reward for the action of the vehicle on the basis of the plurality of individual rewards, and perform reinforcement learning that optimizes the derived reward.

(15): A computer-readable non-transitory storage medium according to another aspect of the present invention stores a program that causes a computer to generate information indicating an action of a vehicle, derive a plurality of individual rewards obtained by evaluating each of a plurality of pieces of information to be evaluated, which include feedback information obtained from a simulator or an actual environment by inputting information based on the information indicating the action of the vehicle to the simulator or the actual environment, derive a reward for the action of the vehicle on the basis of the plurality of individual rewards, and perform reinforcement learning that optimizes the derived reward.

According to the aspects of (1) to (15) described above, it is possible to perform more flexible learning.

