CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to Japanese Patent Application No. 2020-109675 filed on Jun. 25, 2020, incorporated herein by reference in its entirety.

BACKGROUND

1. Technical Field

The disclosure relates to a vehicle control device, a vehicle control system, and a vehicle learning device.

2. Description of Related Art

Japanese Unexamined Patent Application Publication 2000-250602 (JP 2000-250602 A), for example, describes setting an appropriate gear ratio in accordance with the state of a vehicle by reinforcement learning.

SUMMARY

When driving of a transmission that adjusts the gear ration described above is subject to constraint, repeated reinforcement learning may result in the learning results greatly deviating from an appropriate value for when driving of the transmission is not subject to the constraint. This situation is not limited to transmissions, and can occur in drivetrain devices as well.

A vehicle control device according to a first aspect of the disclosure includes: a storage device that stores relation-defining data that is data for defining a relation between a state of a vehicle and an action variable that is a variable relating to operation of a drivetrain device installed in the vehicle; and an executing device configured to acquire the state based on a detection value of an onboard sensor, operate the drivetrain device based on a value of the action variable that is determined based on the relation-defining data and the state acquired by the executing device, derive a reward such that the reward is larger when the state of the drivetrain device based on the state acquired by the executing device satisfies a predetermined criterion than when the state of the drivetrain device based on the state acquired by the executing device does not satisfy the predetermined criterion, perform an updating of the relation-defining data using an updating map of which arguments are the state acquired by the executing device, the value of the action variable used in operating the drivetrain device, and the reward corresponding to the operation and which returns the relation-defining data that is updated such that an expected income regarding the reward calculated when the drivetrain device is operated following the relation-defining data increases, and restrict the updating of the relation-defining data such that an updating amount of the relation-defining data is smaller when the drivetrain device is subject to a predetermined restriction than when the drivetrain device is not subject to the predetermined restriction.

According to the above aspect, execution of updating processing is restricted such that the updating amount is smaller when a restriction regarding driving of the drivetrain device is in effect. Accordingly, a situation in which the relation-defining data greatly changes due to being updated when a restriction regarding driving of the drivetrain device is in effect can be suppressed. Accordingly, in the above configuration, learning results from reinforcement learning can be suppressed from greatly deviating from an appropriate value for operating the drivetrain device in normal operations.

In the above aspect, the executing device may be configured to determine whether an abnormality exists in the drivetrain device; and the drivetrain device may be subject to the predetermined restriction when the executing device determines that the abnormality exists in the drivetrain device.

When reinforcement learning is performed and there is an abnormality in the drivetrain device, there is a possibility that learning results from the reinforcement learning greatly deviate from an appropriate value for operating the drivetrain device in normal operations. According to the above configuration, execution of updating processing is restricted when determination is made that there is an abnormality. Accordingly, learning results from reinforcement learning can be suppressed from greatly deviating from an appropriate value for operating the drivetrain device in normal operations.

In the above aspect, the drivetrain device may include a transmission; and the drivetrain device may be subject to the predetermined restriction when a temperature of operating oil of the transmission is equal to or higher than a high-temperature threshold value.

When reinforcement learning is performed and the temperature of operating oil is excessively high, there is a possibility that learning results from the reinforcement learning greatly deviate from an appropriate value for operating the drivetrain device in a normal temperature range. According to the above configuration, execution of updating processing is restricted when the temperature of the operating oil is equal to or higher than the high-temperature-side threshold value. Accordingly, learning results from reinforcement learning can be suppressed from greatly deviating from an appropriate value for a temperature range in which the temperature of the operating oil is normal.

In the above aspect, the drivetrain device may include the transmission; and the drivetrain device may be subject to the predetermined restriction when the temperature of operating oil of the transmission is equal to or lower than a low-temperature threshold value.

When reinforcement learning is performed and the temperature of operating oil is excessively low, there is a possibility that learning results from the reinforcement learning greatly deviate from an appropriate value for operating the drivetrain device in a normal temperature range. According to the above configuration, execution of updating processing is restricted when the temperature of the operating oil is equal to or lower than the low-temperature-side threshold value. Accordingly, learning results from reinforcement learning can be suppressed from greatly deviating from an appropriate value for a temperature range in which the temperature of the operating oil is normal.

In the above aspect, the updating amount may be zero when the drivetrain device is subject to a predetermined restriction.

According to the above configuration, by setting the updating amount to zero through the restricting processing, a situation where the relation-defining data deviates from appropriate data in normal operations can be sufficiently suppressed, as compared to when the updating amount is made to be smaller but not made to zero.

A vehicle control system according to a second aspect of the disclosure includes: a storage device that stores relation-defining data that is data for defining a relation between a state of a vehicle and an action variable that is a variable relating to operation of a drivetrain device installed in the vehicle; and an executing device including a first executing device that is installed in the vehicle, and a second executing device that is separate from an onboard device, wherein the first executing device is configured to acquire the state based on a detection value of an onboard sensor, and operate the drivetrain device based on a value of the action variable that is determined based on the relation-defining data and the state acquired by the first executing device, at least one of the first executing device or the second executing device is configured to derive a reward such that the reward is larger when the state of the drivetrain device based on the state acquired by the first executing device satisfies a predetermined criterion than when the state of the drivetrain device based on the state acquired by the first executing device does not satisfy the predetermined criterion, the second executing device is configured to perform an updating of the relation-defining data using an updating map of which arguments are the state acquired by the first executing device, the value of the action variable used in operating the drivetrain device, and the reward corresponding to the operation and which returns the relation-defining data that is updated such that an expected income regarding the reward calculated when the drivetrain device is operated following the relation-defining data increases, and the at least one of the first executing device or the second executing device is further configured to restrict the updating of the relation-defining data such that an updating amount of the relation-defining data is smaller when the drivetrain device is subject to a predetermined restriction than when the drivetrain device is not subject to the predetermined restriction.

According to the above configuration, the second executing device executes updating processing, and accordingly computing load on the first executing device can be reduced as compared to when the first executing device executes the updating processing. Note that to say that the second executing device is a separate device from an onboard device means that the second executing device is not an onboard device.

A vehicle control device according to a third aspect of the disclosure includes the first executing device included in the vehicle control system of the second aspect.

A vehicle learning device according to a fourth aspect of the disclosure includes the second executing device included in the vehicle control system of the second aspect.

