BACKGROUND

Mobile computing devices commonly include an array of sensors and communication interfaces that provide information useful for determining a location of the device. Location-based sensors include global positioning system (GPS) sensors and inertial measurements units (IMUs). GPS is capable of providing relatively high-resolution measurements describing the geo-location of a device, but GPS depends on reception of satellite signals and therefore tends to provide more precise location information outdoors, and less precise or no information indoors where there is greater noise in the GPS signal. IMUs measure the motion of a device based on accelerations and forces imparted on the device, but can accumulate error when used to track motions of the device for location purposes over a prolonged period of time.

SUMMARY

This specification discloses computer-based systems, methods, devices, and other techniques for estimating the pose of a device, including estimating the pose based on images captured by a set of image sensors disposed around the device's periphery. As used herein, the term “pose” generally refers to the position of a device in an environment (e.g., x, y, and z coordinates of the device), and can additionally include information about the orientation of the device (e.g., rotation of the device about the x, y, and z axes). The pose of the device can also be mapped to a “location,” which, as used herein, specifies the position of the device with respect to real-world coordinates (e.g., latitude/longitude geolocation coordinates), or otherwise specifies the position of the device in relation to a real-world reference such as a reference location in a map of the environment.

Some implementations include a system that obtains visual data representing at least one image captured by one or more image sensors of a mobile device. The at least one image show an environment of the mobile device, and the one or more image sensors are located at or near respective corners of the mobile device, or at other locations around its periphery. The system processes the visual data to determine a pose of the mobile device. Further, the system can determine a location of the mobile device in the environment based on the pose, and can present an indication of the location of the mobile device in the environment.

Some implementations include a mobile computing device that includes multiple image sensors disposed around a periphery of the device. The mobile device can include one or more processors and one or more computer-readable media having instructions stored thereon that, when executed by the one or more processors, cause the mobile device to perform operations for a computer-implemented method. The method can include obtaining visual data representing images captured by the plurality of image sensors that show an environment of the mobile device, processing the visual data to determine a pose of the mobile device, determining a location of the mobile device in the environment based on the pose of the mobile device, and presenting an indication of the location of the mobile device in the environment.

Additional aspects of the implementations described herein can include one or more computer-readable media having instructions stored thereon that, when executed by one or more processors, cause performance of any of the operations, methods, and/or processes described herein. A mobile device may include the computer-readable media and the one or more processors.

Some implementations of the technology described herein can, in certain instances, realize one or more of the following advantages. First, by locating a set of image sensors around the periphery of a mobile device, the sensors may provide a wide field of view that covers a substantial portion of the environment around the device (e.g., up to 360-degree vision in all directions around the device). Second, by locating respective image sensors at the corners of a mobile device, each sensor may provide a wide field of view that covers both a lateral portion of the environment and an upper or lower portion of the environment around the device. Additionally, because users tend to grasp their devices along the sides (e.g., the lateral edges of the device), all or some of the corners tend to be exposed. The corner-located image sensors reduce occurrences of occlusion of the sensors' fields of view, and allows the device to perform pose estimation during the normal course of a user's interaction with the device. Fourth, by providing image sensors around the periphery of the device, the device may acquire images that allow it to determine estimates of the pose and location of the device based on analysis of features in the images. Vision-enhanced pose and location estimation can, for example, be used to rectify errors in estimations derived from accelerometer, gyroscope, and/or magnetometer signals. Fifth, by processing visual data, the device may determine its pose and location with high accuracy even when the device has no ability or degraded ability to receive signals from external sources that would indicate its location, such as global positioning system (GPS) or other wireless signals. Thus, the device may process visual data based on images captured by image sensors located around a periphery of the device to estimate its pose and location where GPS is not available, such as in an indoor environment. Sixth, the imaging system for capturing images of the environment and generating visual data can facilitate pose and estimation location with relatively low power, thereby having only a small impact on the battery life of a mobile device. The imaging system consumes relatively low power because an active receiver is unnecessary to detect incoming wireless signals (e.g., GPS signals), the images captured may be relatively low-resolution, and the frequency at which images of the environment are automatically captured may be adjusted according to power management policies (e.g., the frequency can be reduced when the battery power is lower). Seventh, the imaging system may generate visual data representing the environment of the mobile device while preserving privacy interests of persons in the environment by capturing low-resolution images, degrading the images, and/or abstracting the images before they are made available to processes and components outside of the imaging system.

The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features and advantages will be apparent from the description and drawings, and from the claims.

