TECHNICAL FIELD

This disclosure relates to vehicle controllers for agricultural and industrial applications.

BACKGROUND

Autonomous vehicles have been made for automated navigation on roads and obstacle courses. These autonomous vehicles uses sensors, such as cameras and lidar, to sense objects in their environment. For example, an autonomous vehicle may generate motion plans and affect those motion plans by controlling wheels of the vehicles via control of an engine and transmission system, a braking system, and a steering system.

SUMMARY

Disclosed herein are implementations of vehicle controllers for agricultural and industrial applications.

In a first aspect, the subject matter described in this specification can be embodied in systems that include one or more motion sensors configured to detect motion of a vehicle; actuators configured to control motion of the vehicle and to control operation of an implement, wherein the implement is connected to the vehicle and configured to selectively perform an operation in a vicinity of the vehicle; and a processing apparatus configured to: access a map data structure storing a map representing locations of physical objects in a geographic area; access a path data structure that includes a sequence of waypoints that each specify a position within the map, wherein a waypoint in the sequence of waypoints includes implement control data for operating the implement at a location corresponding to the position of the waypoint; access motion sensor data captured using the one or more motion sensors; determine, based on the motion sensor data, an estimate of a current location of the vehicle; control one or more of the actuators to cause the vehicle to move from the current location of the vehicle to the location corresponding to the position of the waypoint; detect, based on the motion sensor data, when the vehicle arrives at the location corresponding to the position of the waypoint; and responsive to the vehicle arriving at the location corresponding to the position of the waypoint, control one or more of the actuators, based on the implement control data of the waypoint, to cause the implement to perform the operation.

In the first aspect, the implement may be configured to selectively deposit input in a vicinity of the vehicle, and the processing apparatus may be configured to: responsive to the vehicle arriving at the location corresponding to the position of the waypoint, control one or more of the actuators, based on the implement control data of the waypoint, to cause the implement to deposit an input in a vicinity of the vehicle. In the first aspect, the implement may be configured to selectively collect output in a vicinity of the vehicle, and the processing apparatus may be configured to: responsive to the vehicle arriving at the location corresponding to the position of the waypoint, control one or more of the actuators, based on the implement control data of the waypoint, to cause the implement to collect an output in a vicinity of the vehicle. In the first aspect, the processing apparatus may be configured to: determine localization data based on motion data from the one or more motion sensors collected while a user manually controls the vehicle and the implement, wherein the localization data includes a sequence of positions within the map; record implement control data associated with positions in the localization data; and generate the path data structure based on the localization data and the associated implement control data. In the first aspect, the processing apparatus may be configured to: present the map to a user in a user interface that enables the user to draw a path for the vehicle on the map; and generate the path data structure based on the drawn path. In the first aspect, the processing apparatus may be configured to: receive boundary data specifying an area within the map; and generate the path data structure based on the map and the boundary data. The first aspect may include one or more image sensors connected to the vehicle, and the processing apparatus may be configured to: receive image data, captured using the one or more image sensors, depicting one or more plants in the vicinity of the vehicle; detect the one or more plants based on the image data; responsive to detecting the one or more plants, adjust implement control data; and control one or more of the actuators, based on the adjusted implement control data, to cause the implement to deposit an input on the one or more plants. In the first aspect, the one or more image sensors may include a normalized difference vegetation index camera connected to the vehicle, and the processing apparatus may be configured to: receive normalized difference vegetation index data, captured using the normalized difference vegetation index camera, for the one or more plants, wherein adjusting the implement control data includes adjusting an amount of input to be deposited on the one or more plants based on the normalized difference vegetation index data. The first aspect may include a distance sensor connected to the vehicle, and the processing apparatus may be configured to: access current point cloud data captured using the distance sensor; compare the current point cloud data to point cloud data of the map that is selected based on the motion sensor data; and determine the estimate of the current location of the vehicle based on the comparison of the current point cloud data to point cloud data of the map. In the first aspect, the processing apparatus may be configured to: detect a furrow in the ground based on the current point cloud data; match the detected furrow with a furrow represented in the map; and determine the estimate of the current location of the vehicle based on a current position in relation to the detected furrow. In the first aspect, the processing apparatus may be configured to: detect a crop row based on the current point cloud data; match the detected crop row with a crop row represented in the map; and determine the estimate of the current location of the vehicle based on a current position in relation to the detected crop row. In the first aspect, the processing apparatus may be configured to: filter the current point cloud data to consider only points in a zone of interest when comparing the current point cloud data to point cloud data of the map, where in the zone of interest is limited to a range of heights and a maximum distance from the vehicle. In the first aspect, the vehicle may be a tractor and the implement may be a sprayer. In the first aspect, the vehicle may be a tractor and the implement may be a tillage implement.

In a second aspect, the subject matter described in this specification can be embodied in systems that include a distance sensor connected to a vehicle; actuators configured to control motion of the vehicle; and a processing apparatus configured to: access a map data structure storing a map representing locations of physical objects in a geographic area; access current point cloud data captured using the distance sensor; detect a crop row based on the current point cloud data; match the detected crop row with a crop row represented in the map; determine an estimate of a current location of the vehicle based on a current position in relation to the detected crop row; and control one or more of the actuators to cause the vehicle to move from the current location of the vehicle to a target location.

In the second aspect, the processing apparatus may be configured to: detect a furrow in the ground based on the current point cloud data; match the detected furrow with a furrow represented in the map; and determine the estimate of the current location of the vehicle based on a current position in relation to the detected furrow. In the second aspect, the processing apparatus may be configured to: filter the current point cloud data to consider only points in a zone of interest when comparing the current point cloud data to point cloud data of the map to match the detected crop row with the crop row represented in the map, where in the zone of interest is limited to a range of heights and a maximum distance from the vehicle. The second aspect may include one or more image sensors connected to the vehicle; and actuators configured to control operation of an implement, wherein the implement is connected to the vehicle and configured to selectively perform an operation in a vicinity of the vehicle. In the second aspect, the processing apparatus may be configured to: receive image data, captured using the one or more image sensors, depicting one or more plants in the vicinity of the vehicle; detect the one or more plants based on the image data; responsive to detecting the one or more plants, adjust implement control data; and control one or more of the actuators, based on the adjusted implement control data, to cause the implement to deposit an input on the one or more plants. In the second aspect, the one or more image sensors may include a normalized difference vegetation index camera connected to the vehicle, and the processing apparatus may be configured to: receive normalized difference vegetation index data, captured using the normalized difference vegetation index camera, for the one or more plants, wherein adjusting the implement control data includes adjusting an amount of input to be deposited on the one or more plants based on the normalized difference vegetation index data.

In a third aspect, the subject matter described in this specification can be embodied in systems that include one or more image sensors connected to a vehicle; actuators configured to control operation of an implement, wherein the implement is connected to the vehicle and configured to selectively perform an operation in a vicinity of the vehicle; and a processing apparatus configured to: receive image data, captured using the one or more image sensors, depicting one or more plants in a vicinity of the vehicle; detect the one or more plants based on the image data; responsive to detecting the one or more plants, adjust implement control data; and control one or more of the actuators, based on the adjusted implement control data, to cause the implement to perform the operation on the one or more plants.

In the third aspect, the one or more image sensors may include a normalized difference vegetation index camera connected to the vehicle, and the processing apparatus may be configured to: receive normalized difference vegetation index data, captured using the normalized difference vegetation index camera, for the one or more plants, wherein adjusting the implement control data includes adjusting an amount of input to be deposited on the one or more plants based on the normalized difference vegetation index data. In the third aspect, the implement may be configured to selectively deposit input in a vicinity of the vehicle, and the processing apparatus may be configured to: control one or more of the actuators, based on the adjusted implement control data, to cause the implement to deposit an input on the one or more plants. In the third aspect, the implement may be configured to selectively collect output in a vicinity of the vehicle, and the processing apparatus may be configured to: control one or more of the actuators, based on the adjusted implement control data, to cause the implement to collect an output from the one or more plants.

In a fourth aspect, the subject matter described in this specification can be embodied in methods that include accessing motion sensor data captured using one or more motion sensors configured to detect a motion of a vehicle; determining, based on the motion sensor data, an estimate of a current location of the vehicle; accessing a map data structure storing a map representing locations of physical objects in a geographic area; accessing a path data structure that includes a sequence of waypoints that each specify a position within the map, wherein a waypoint in the sequence of waypoints includes implement control data for operating an implement at a location corresponding to the position of the waypoint, wherein the implement is connected to the vehicle and configured to selectively perform an operation in a vicinity of the vehicle; controlling one or more actuators to cause the vehicle to move from the current location of the vehicle to the location corresponding to the position of the waypoint; detecting, based on the motion sensor data, when the vehicle arrives at the location corresponding to the position of the waypoint; and, responsive to the vehicle arriving at the location corresponding to the position of the waypoint, controlling, based on the implement control data of the waypoint, the implement to perform the operation.

In the fourth aspect, the implement may be configured to selectively deposit input in a vicinity of the vehicle, and the method may include: responsive to the vehicle arriving at the location corresponding to the position of the waypoint, controlling, based on the implement control data of the waypoint, the implement to deposit an input in a vicinity of the vehicle. In the third aspect, the implement may be configured to selectively collect output in a vicinity of the vehicle, and the method may include: responsive to the vehicle arriving at the location corresponding to the position of the waypoint, controlling, based on the implement control data of the waypoint, the implement to collect an output in a vicinity of the vehicle. The third aspect may include determining localization data based on motion data from the one or more motion sensors collected while a user manually controls the vehicle and the implement, wherein the localization data includes a sequence of positions within the map; recording implement control data associated with positions in the localization data; and generating the path data structure based on the localization data and the associated implement control data. The third aspect may include presenting the map to a user in a user interface that enables the user to draw a path for the vehicle on the map; and generating the path data structure based on the drawn path. The third aspect may include receiving boundary data specifying an area within the map; and generating the path data structure based on the map and the boundary data. The third aspect may include receiving image data, captured using one or more image sensors connected to the vehicle, depicting one or more plants in a vicinity of the vehicle; detecting the one or more plants based on the image data; responsive to detecting the one or more plants, adjusting implement control data; and controlling, based on the adjusted implement control data, the implement to deposit an input on the one or more plants. In the third aspect, the one or more image sensors may include a normalized difference vegetation index camera connected to the vehicle, and the method may include: receiving normalized difference vegetation index data, captured using the normalized difference vegetation index camera, for the one or more plants, wherein adjusting the implement control data includes adjusting an amount of input to be deposited on the one or more plants based on the normalized difference vegetation index data. The third aspect may include accessing current point cloud data captured using a distance sensor mounted to the vehicle; comparing the current point cloud data to point cloud data of the map that is selected based on the motion sensor data; and determining the estimate of the current location of the vehicle based on the comparison of the current point cloud data to point cloud data of the map. The third aspect may include detecting a furrow in the ground based on the current point cloud data; matching the detected furrow with a furrow represented in the map; and determining the estimate of the current location of the vehicle based on a current position in relation to the detected furrow. The third aspect may include detecting a crop row based on the current point cloud data; matching the detected crop row with a crop row represented in the map; and determining the estimate of the current location of the vehicle based on a current position in relation to the detected crop row. The third aspect may include filtering the current point cloud data to consider only points in a zone of interest when comparing the current point cloud data to point cloud data of the map, where in the zone of interest is limited to a range of heights and a maximum distance from the vehicle. In the fourth aspect, the vehicle may be a tractor and the implement may be a sprayer. In the fourth aspect, the vehicle may be a tractor and the implement may be a tillage implement.

In a fifth aspect, the subject matter described in this specification can be embodied in methods that include accessing a map data structure storing a map representing locations of physical objects in a geographic area; accessing current point cloud data captured using a distance sensor connected to a vehicle; detecting a crop row based on the current point cloud data; matching the detected crop row with a crop row represented in the map; determining an estimate of a current location of the vehicle based on a current position in relation to the detected crop row; and controlling one or more actuators to cause the vehicle to move from the current location of the vehicle to a target location.

The fifth aspect may include detecting a furrow in the ground based on the current point cloud data; matching the detected furrow with a furrow represented in the map; and determining the estimate of the current location of the vehicle based on a current position in relation to the detected furrow. The fifth aspect may include filtering the current point cloud data to consider only points in a zone of interest when comparing the current point cloud data to point cloud data of the map to match the detected crop row with the crop row represented in the map, where in the zone of interest is limited to a range of heights and a maximum distance from the vehicle. The fifth aspect may include receiving image data, captured using one or more image sensors connected to the vehicle, depicting one or more plants in a vicinity of the vehicle; detecting the one or more plants based on the image data; responsive to detecting the one or more plants, adjusting implement control data; and controlling, based on the adjusted implement control data, an implement to deposit an input on the one or more plants, wherein the implement is connected to the vehicle and configured to selectively deposit input in a vicinity of the vehicle. In the fifth aspect, the one or more image sensors may include a normalized difference vegetation index camera connected to the vehicle, and the method may include: receiving normalized difference vegetation index data, captured using the normalized difference vegetation index camera, for the one or more plants, wherein adjusting the implement control data includes adjusting an amount of input to be deposited on the one or more plants based on the normalized difference vegetation index data.

In a sixth aspect, the subject matter described in this specification can be embodied in methods that include receiving image data, captured using one or more image sensors connected to a vehicle, depicting one or more plants in a vicinity of the vehicle; detecting the one or more plants based on the image data; responsive to detecting the one or more plants, adjusting implement control data; and controlling, based on the adjusted implement control data, an implement connected to the vehicle to perform an operation on the one or more plants.

In the sixth aspect, the one or more image sensors may include a normalized difference vegetation index camera connected to the vehicle, and the method may include: receiving normalized difference vegetation index data, captured using the normalized difference vegetation index camera, for the one or more plants, wherein adjusting the implement control data includes adjusting an amount of input to be deposited on the one or more plants based on the normalized difference vegetation index data. In the sixth aspect, the implement may be configured to selectively deposit input in a vicinity of the vehicle, and the method may include: controlling, based on the adjusted implement control data, the implement to deposit an input on the one or more plants. In the sixth aspect, the implement may be configured to selectively collect output in a vicinity of the vehicle, and the method may include: controlling, based on the adjusted implement control data, the implement to collect an output from the one or more plants.

In a seventh aspect, the subject matter described in this specification can be embodied in a non-transitory computer-readable storage medium storing executable instructions that, when executed by a processor, facilitate performance of operations, including accessing motion sensor data captured using one or more motion sensors configured to detect a motion of a vehicle; determining, based on the motion sensor data, an estimate of a current location of the vehicle; accessing a map data structure storing a map representing locations of physical objects in a geographic area; accessing a path data structure that includes a sequence of waypoints that each specify a position within the map, wherein a waypoint in the sequence of waypoints includes implement control data for operating an implement at a location corresponding to the position of the waypoint, wherein the implement is connected to the vehicle and configured to selectively perform an operation in a vicinity of the vehicle; controlling one or more actuators to cause the vehicle to move from the current location of the vehicle to the location corresponding to the position of the waypoint; detecting, based on the motion sensor data, when the vehicle arrives at the location corresponding to the position of the waypoint; and, responsive to the vehicle arriving at the location corresponding to the position of the waypoint, controlling, based on the implement control data of the waypoint, the implement to perform the operation.

In a eighth aspect, the subject matter described in this specification can be embodied in a non-transitory computer-readable storage medium storing executable instructions that, when executed by a processor, facilitate performance of operations, including accessing a map data structure storing a map representing locations of physical objects in a geographic area; accessing current point cloud data captured using a distance sensor connected to a vehicle; detecting a crop row based on the current point cloud data; matching the detected crop row with a crop row represented in the map; determining an estimate of a current location of the vehicle based on a current position in relation to the detected crop row; and controlling one or more actuators to cause the vehicle to move from the current location of the vehicle to a target location.

In a ninth aspect, the subject matter described in this specification can be embodied in a non-transitory computer-readable storage medium storing executable instructions that, when executed by a processor, facilitate performance of operations, including receiving image data, captured using one or more image sensors connected to a vehicle, depicting one or more plants in a vicinity of the vehicle; detecting the one or more plants based on the image data; responsive to detecting the one or more plants, adjusting implement control data; and controlling, based on the adjusted implement control data, an implement connected to the vehicle to perform an operation on the one or more plants.

