TECHNICAL FIELD

This disclosure relates to navigating mobile robots.

BACKGROUND

As technology advances, various types of robotic devices are being created for performing a variety of functions that may assist users. Robotic devices may be used for applications involving material handling, transportation, welding, assembly, and dispensing, among others. Over time, the manner in which these robotic systems operate is becoming more intelligent, efficient, and intuitive. As robotic systems become increasingly prevalent in numerous aspects of modern life, it is desirable for robotic systems to be easy and intuitive for people to operate and navigate.

SUMMARY

One aspect of the disclosure provides a method for controlling a robot. The method includes receiving, at data processing hardware of an operator device, image data from at least one image sensor. The image data corresponds to an environment about the robot. The method also includes executing, by the data processing hardware, a graphical user interface (GUI) for display on a screen of the operator device. The GUI is configured to display a scene of the environment about the robot based on the image data and receive an input indication indicating selection of a pixel location within the scene of the environment about the robot. The method also includes determining, by the data processing hardware, a pointing vector V based on the selection of the pixel location. The pointing vector represents a direction of travel for navigating the robot in the environment. The method also includes transmitting, by the data processing hardware, a waypoint command W to the robot. The waypoint command W when received by the robot causes the robot to navigate to a target location T. The target location is based on an intersection between the pointing vector V and a terrain estimate G of the robot. The terrain estimate G may include a ground plane estimate. For simplicity, the present disclosure will refer to the “terrain estimate” as the “ground plane estimate”.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the scene of the environment about the robot includes any one of: a forward scene of the environment based on the image data, the image data captured by a forward-left camera and a forward-right camera disposed on the robot; a left scene of the of the environment based on the image data, the image data captured by a left camera disposed on the robot; a right scene of the environment based on the image data, the image data captured by a right camera disposed on the mobile robot; an aft scene of the environment based on the image data, the aft scene captured by an aft camera disposed on the robot; or a top-down scene of the robot based on the image data, the image data captured by a payload camera, the forward-left camera, the forward-right camera, the left camera, the right camera, and the aft camera.

In some examples, the method further includes determining, by the data processing hardware, whether the intersection between the pointing vector and the terrain estimate is in front of the robot, and when the intersection between the pointing vector and the terrain estimate is in front of the robot, identifying, by the data processing hardware, the intersection as the target location. In these examples, the method may further include, when the intersection between the pointing vector and the terrain estimate is behind the robot: reflecting, by the data processing hardware, the intersection to determine a reflected intersection in front of the robot; and identifying, by the data processing hardware, the reflected intersection as the target location. Additionally or alternatively, in these examples, the method may further include, prior to identifying the intersection as the target location: determining, by the data processing hardware, that a first distance between a current position of the robot and the intersection between the pointing vector and the terrain estimate dissatisfies a threshold distance; and moving, by the data processing hardware, the intersection to an intermediate location closer to the robot to form a moved intersection, wherein a second distance between the current position of the robot and the moved intersection satisfies the threshold distance.

The robot may be configured to determine whether the target location is located behind an obstacle. Here, the robot is configured to autonomously navigate to the target location when the target location is not located behind the obstacle. However, when the target location is located behind the obstacle, the robot is configured to determine whether maneuvering around the obstacle is feasible, and when maneuvering around the obstacle is feasible, autonomously navigate the robot around the obstacle and to the target location. Additionally, when the target location is located behind the obstacle and when maneuvering around the obstacle is not feasible, the robot may be further configured to prevent the robot from autonomously navigating to the target location.

In some implementations, the at least one image sensor includes one or more fisheye cameras and one or more infrared cameras. The at least one image sensor may be disposed on the robot and the operator device may be in communication with the image sensor via a network.

In some examples, the graphical user interface is further configured to: receive a rotation input to rotate a field of view of the environment about the robot in a direction away from a current scene displayed in the graphical user interface; and display a preview scene by rotating the field of view of the environment about the robot in the direction away from the current scene. The graphical user interface may be configured to display the preview scene without requiring physical movement by the robot and/or the graphical user interface may be configured to receive the rotation input in response to receiving an input indication indicating selection of a rotation graphic displayed in the graphical user interface. In additional examples, the rotation of the field of view of the environment about the robot in the direction away from the current scene simulates the robot executing a turning maneuver in the direction away from the current scene and toward the preview scene.

Another aspect of the disclosure provides for a system for operating a robot. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform operations that include receiving image data from at least one image sensor. The image data corresponds to an environment about the robot. The operations also include executing a graphical user interface (GUI) for display on a screen of the operator device. The GUI is configured to display a scene of the environment about the robot based on the image data and receive an input indication indicating selection of a pixel location within the scene of the environment about the robot. The operations also include determining a pointing vector V based on the selection of the pixel location. The pointing vector represents a direction of travel for navigating the robot in the environment. The operations also include transmitting a waypoint command W to the robot. The waypoint command W when received by the robot causes the robot to navigate to a target location T. The target location is based on an intersection between the pointing vector V and a terrain estimate G of the robot.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the scene of the environment about the robot includes any one of: a forward scene of the environment based on the image data, the image data captured by a forward-left camera and a forward-right camera disposed on the robot; a left scene of the of the environment based on the image data, the image data captured by a left camera disposed on the robot; a right scene of the environment based on the image data, the image data captured by a right camera disposed on the mobile robot; an aft scene of the environment based on the image data, the aft scene captured by an aft camera disposed on the robot; or a top-down scene of the robot based on the image data, the image data captured by a payload camera, the forward-left camera, the forward-right camera, the left camera, the right camera, and the aft camera.

In some examples, the operations further include determining whether the intersection between the pointing vector and the terrain estimate is in front of the robot, and when the intersection between the pointing vector and the terrain estimate is in front of the robot, identifying the intersection as the target location. In these examples, the operations may further include, when the intersection between the pointing vector and the terrain estimate is behind the robot: reflecting the intersection to determine a reflected intersection in front of the robot; and identifying the reflected intersection as the target location. Additionally or alternatively, in these examples, the operations may further include, prior to identifying the intersection as the target location: determining that a first distance between a current position of the robot and the intersection between the pointing vector and the terrain estimate dissatisfies a threshold distance; and moving the intersection to an intermediate location closer to the robot to form a moved intersection, wherein a second distance between the current position of the robot and the moved intersection satisfies the threshold distance.

The robot may be configured to determine whether the target location is located behind an obstacle. Here, the robot is configured to autonomously navigate to the target location when the target location is not located behind the obstacle. However, when the target location is located behind the obstacle, the robot is configured to determine whether maneuvering around the obstacle is feasible, and when maneuvering around the obstacle is feasible, autonomously navigate the robot around the obstacle and to the target location. Additionally, when the target location is located behind the obstacle and when maneuvering around the obstacle is not feasible, the robot may be further configured to prevent the robot from autonomously navigating to the target location. The robot may include a quadruped robot.

In some implementations, the at least one image sensor includes one or more fisheye cameras and one or more infrared cameras. The at least one image sensor may be disposed on the robot and the operator device may be in communication with the image sensor via a network.

In some examples, the graphical user interface is further configured to: receive a rotation input to rotate a field of view of the environment about the robot in a direction away from a current scene displayed in the graphical user interface; and display a preview scene by rotating the field of view of the environment about the robot in the direction away from the current scene. The graphical user interface may be configured to display the preview scene without requiring physical movement by the robot and/or the graphical user interface may be configured to receive the rotation input in response to receiving an input indication indicating selection of a rotation graphic displayed in the graphical user interface. In additional examples, the rotation of the field of view of the environment about the robot in the direction away from the current scene simulates the robot executing a turning maneuver in the direction away from the current scene and toward the preview scene.

Another aspect of the disclosure provides a method for navigating a mobile robot. The method includes receiving, at data processing hardware, a pixel location in a scene of a robotic environment captured by a camera on a mobile robot and determining, by the data processing hardware, a pointing vector from the pixel location and calibration information of the camera. The pixel location being received is selected by an operator of the mobile robot. The method further includes combining, by the data processing hardware, the pointing vector with a terrain estimate of the mobile robot to form an intersection. The intersection includes a physical location in the robotic environment. The method further includes transmitting, by the data processing hardware, the physical location to the mobile robot causing the mobile robot to navigate to the physical location in the robotic environment.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the scene is any one of i) a forward scene of the robotic environment captured by a forward-left camera and a forward-right camera on the mobile robot, ii) a left scene of the robotic environment captured by a left camera on the mobile robot, iii) a right scene of the robotic environment captured by a right camera on the mobile robot, iv) an aft scene of the robotic environment captured by an aft camera on the mobile robot, and v) a top-down scene of the robotic environment captured by a hypothetical camera positioned above the mobile robot.

In some examples of the method, combining the pointing vector with the terrain estimate to form the intersection includes determining, by the data processing hardware, whether a distance between the mobile robot and the intersection is greater than a distance threshold of the mobile robot. These examples of the method further include when the distance is greater than the distance threshold of the mobile robot, moving, by the data processing hardware, the intersection closer to the mobile robot to form a moved intersection such that a shortened distance between the mobile robot and the moved intersection is less than or equal to the distance threshold.

In other examples of the method, combining the pointing vector with the terrain estimate to form the intersection includes determining, by the data processing hardware, whether the intersection is behind the mobile robot. These examples of the method further include when the intersection is behind the mobile robot, reflecting, by the data processing hardware, the intersection in front of the mobile robot to form a reflected intersection. Here, the reflected intersection includes another physical location in the robotic environment for the mobile robot to navigate to. Optionally, some other examples of the method further include determining, by the data processing hardware, whether a distance between the mobile robot and the reflected intersection is greater than a distance threshold of the mobile robot. These other examples of the method further include when the distance is greater than the distance threshold of the mobile robot, moving, by the data processing hardware, the reflected intersection closer to the mobile robot to form a moved intersection such that a shortened distance between the mobile robot and the moved intersection is less than or equal to the distance threshold of the mobile robot.

In some examples of the method, the scene of the robotic environment includes a first scene of the robotic environment captured by a first camera on the mobile robot and at least one second scene of the robotic environment different than the first scene of the robotic environment. The at least one second scene of the robotic environment captured by at least one second camera on the mobile robot is different than the first camera on the mobile robot. In these examples, the method further includes displaying, by the data processing hardware, the first scene of the robotic environment together with the at least one second scene of the robotic environment. The method further includes in response to the operator changing between the first scene of the robotic environment and the at least one second scene of the robotic environment, changing, by the data processing hardware, a direction in which the mobile robot navigates.

In other examples of the method, the scene is a first-person view of the robotic environment and the method further includes displaying, by the data processing hardware, the scene together with and a top-down scene of the robotic environment captured by a hypothetical camera positioned above the mobile robot. The method further includes in response to a pixel location selection in one of the displayed scenes, mirroring, by the data processing hardware, the pixel location selection in another one of the displayed scenes.

In some examples of the method, transmitting the physical location in the robotic environment includes transmitting a position command with the physical location to the mobile robot. Optionally, the camera may have a fisheye camera and two infrared cameras. Additionally, the mobile robot may have four legs.

Another aspect of the disclosure provides for a system for navigating a mobile robot. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform operations including receiving a pixel location in a scene of a robotic environment captured by a camera on a mobile robot and determining a pointing vector from the pixel location and calibration information of the camera. The pixel location is selected by an operator of the mobile robot. The operations performed by the data processing hardware further include combining the pointing vector with a terrain estimate of the mobile robot to form an intersection. The intersection includes a physical location in the robotic environment. The operations performed by the data processing hardware further include transmitting the physical location to the mobile robot causing the mobile robot to navigate to the physical location in the robotic environment.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the scene is any one of i) a forward scene of the robotic environment captured by a forward-left camera and a forward-right camera on the mobile robot, ii) a left scene of the robotic environment captured by a left camera on the mobile robot, iii) a right scene of the robotic environment captured by a right camera on the mobile robot, iv) an aft scene of the robotic environment captured by an aft camera on the mobile robot, and v) a top-down scene of the robotic environment captured by a hypothetical camera positioned above the mobile robot.

In some examples of the system, the operation of combining the pointing vector with the terrain estimate to form the intersection includes determining whether a distance between the mobile robot and the intersection is greater than a distance threshold of the mobile robot. The operations further include when the distance is greater than the distance threshold of the mobile robot, moving the intersection closer to the mobile robot to form a moved intersection such that a shortened distance between the mobile robot and the moved intersection is less than or equal to the distance threshold.

In other examples of the system, the operation of combining the pointing vector with the terrain estimate to form the intersection includes determining whether the intersection is behind the mobile robot. The operations further include when the intersection is behind the mobile robot, reflecting the intersection in front of the mobile robot to form a reflected intersection. Here, the reflected intersection includes another physical location in the robotic environment for the mobile robot to navigate to. Optionally, in some other examples of the system, the operations further include determining whether a distance between the mobile robot and the reflected intersection is greater than a distance threshold of the mobile robot. The operation further include when the distance is greater than the distance threshold of the mobile robot, moving, by the data processing hardware, the reflected intersection closer to the mobile robot to form a moved intersection such that a shortened distance between the mobile robot and the moved intersection is less than or equal to the distance threshold of the mobile robot.

In some examples of the system, the scene of the robotic environment includes a first scene of the robotic environment captured by a first camera on the mobile robot and at least one second scene of the robotic environment different than the first scene of the robotic environment. The at least one second scene of the robotic environment captured by at least one second camera on the mobile robot is different than the first camera on the mobile robot. In these examples of the system, the operations further includes displaying the first scene of the robotic environment together with the at least one second scene of the robotic environment. The operations further include in response to the operator changing between the first scene of robotic the environment and the at least one second scene of the robotic environment, changing a direction in which the mobile robot navigates.

In other examples of the system, the scene is a first-person view of the robotic environment and the operations further include displaying the scene together with and a top-down scene of the robotic environment captured by a hypothetical camera positioned above the mobile robot. The operation further includes in response to a pixel location selection in one of the displayed scenes, mirroring the pixel location selection in another one of the displayed scenes.

In some examples of the system, the operation of transmitting the physical location in the robotic environment includes transmitting a position command with the physical location to the mobile robot. Optionally, the camera may have a fisheye camera and two infrared cameras. Additionally, the mobile robot may have four legs.

Yet another aspect of the disclosure provides for a method for “pre-rotating” a mobile robot. The method includes receiving, at data processing hardware, a real image of a forward view of a robotic environment captured by a one or more real cameras disposed on a mobile robot. Where the mobile robot is in a forward orientation and facing a forward direction. The method further includes determining, by the data processing hardware, a preview angle by which the mobile robot will rotate when turning from the forward orientation to a turned orientation, where the mobile robot is facing a rightward direction or a leftward direction. The method further includes for each point on a virtual rectangle projected by the hypothetical scene camera, determining, by the data processing hardware, whether a respective point corresponds to a pixel in the real image and displaying, by the data processing hardware, a preview scene includes the corresponding pixels. The preview scene being displayed is a view of the robotic environment that is turned by the preview angle in either the rightward direction or the leftward direction without the mobile robot actually turning from the forward orientation to the turned orientation.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the method further includes when the respective point on the virtual rectangle corresponds to the pixel in the real image, coloring in, by the data processing hardware, the respective point with an image color from the corresponding pixel. The method further includes when the respective point on the virtual rectangle does not correspond to any pixel in the real image, coloring in, by the data processing hardware, the respective point black. The method further includes rendering, by the data processing hardware, the preview scene based on the colored in virtual rectangle.

In other examples of the method, determining the preview angle includes receiving, by the data processing hardware, a virtual velocity corresponding to user input provided by an operator for controlling the mobile robot to turn from the forward orientation to the turned orientation. These examples of the method further include integrating, by the data processing hardware, the virtual velocity over time to generate the preview angle.

Some examples of the method further include starting, by the data processing hardware, a countdown timer in response to an operator providing user input for controlling the mobile robot to turn from the forward orientation to the turned orientation. These examples of the method further include before the countdown timer expires, determining continuously, by the data processing hardware, the preview angle to which to turn the view of the robotic environment based on the user input provided by the operator. These examples of the method further include when the countdown timer expires, transmitting, by the data processing hardware, an angle of rotation that is equal to the preview angle to the mobile robot causing the mobile robot to turn by the angle of rotation from the forward orientation to the turned orientation.

In other examples of method, the real image of the forward view of the robotic environment is captured by a forward-left camera and a forward-right camera disposed on the mobile robot. Optionally, the camera may have a fisheye camera and two infrared cameras. Additionally, the mobile robot may have four legs.

Still yet another aspect of the disclosure provides for a system for “pre-rotating” a mobile robot. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stories instructions that when executed on the data processing hardware cause the data processing hardware to perform operations including receiving a real image of a forward view of a robotic environment captured by a one or more real cameras disposed on a mobile robot, where the mobile robot is in a forward orientation and facing a forward direction. The operations performed by the data processing hardware further include determining a preview angle by which the mobile robot will rotate when turning from the forward orientation to a turned orientation, where the mobile robot is facing a rightward direction or a leftward direction. The operations performed by the data processing hardware further include for each point on a virtual rectangle projected by the hypothetical scene camera, determining whether a respective point corresponds to a pixel in the real image and displaying a preview scene including the corresponding pixels. The preview scene being a view of the robotic environment that is turned by the preview angle in either the rightward direction or the leftward direction without the mobile robot actually turning from the forward orientation to the turned orientation.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the operations further include when the respective point on the virtual rectangle corresponds to the pixel in the real image, coloring in the respective point with an image color from the corresponding pixel. The operations further include when the respective point on the virtual rectangle does not correspond to any pixel in the real image, coloring in, by the data processing hardware, the respective point black. The operations further rendering hardware the preview scene based on the colored in virtual rectangle.

In other examples of the system, the operation of determining the preview angle includes receiving a virtual velocity corresponding to user input provided by an operator for controlling the mobile robot to turn from the forward orientation to the turned orientation. These examples of the system further include integrating the virtual velocity over time to generate the preview angle.

In some examples of the system, the operations further include starting a countdown timer in response to an operator providing user input for controlling the mobile robot to turn from the forward orientation to the turned orientation. In these examples of the system, the operations further include before the countdown timer expires, determining continuously the preview angle to which to turn the view of the robotic environment based on the user input provided by the operator. In these examples of the system, the operations further include when the countdown timer expires, transmitting an angle of rotation that is equal to the preview angle to the mobile robot causing the mobile robot to turn by the angle of rotation from the forward orientation to the turned orientation.

In other examples of system, the real image of the forward view of the robotic environment is captured by a forward-left camera and a forward-right camera disposed on the mobile robot. Optionally, the camera may have a fisheye camera and two infrared cameras. Additionally, the mobile robot may have four legs.

