FIELD OF THE INVENTION

The present invention concerns determining the location of a mobile device. More particularly, but not exclusively, the invention concerns using neural networks to determine the location of a mobile device from images captured by a camera of the mobile device.

The invention is particularly, but not exclusively, applicable where the mobile device is an autonomous robot. However, the invention is also applicable to other types of mobile and wearable devices, such as mobile phones, smart watches and the like.

“Location” as discussed herein may refer to absolute location, such as that a mobile device is at a position on the Earth defined by latitude and longitude, and may also refer to relative location with respect to another position (e.g. a mobile device is a distance and direction from an initial starting position). Determination of location also frequently includes determination of orientation, in absolute terms with respect to the Earth's magnetic field for example, and in relative terms as rotation of a certain amount with respect to an initial orientation.

BACKGROUND OF THE INVENTION

It is desirable to be able to determine the location of mobile devices where no GPS signal is available. This is particularly the case for autonomous robots, to allow accurate navigation. A known approach is to use images from a camera in order to determine location. However, such systems tend to require very precise calibration of the camera if location is to be reliably determined. Traditional visual odometry techniques include sparse methods (comprising several steps, including detection and matching of features, motion estimation and optimisation), and direct methods (comprising steps of motion estimation and optimisation). Such techniques tend to require precise camera calibration, and often fail in texture-poor environments (i.e. where there are with few features) or when a camera capturing images is rotated rapidly. In addition, while such systems are usually able to determine the shape of a path of travel, they are usually unable to estimate scale, i.e. the actual distance travelled.

Alternatively, it is known to use a neural network to process the images from a camera in order to determine location. Mohanty et al, DeepVO: A Deep Learning approach for Monocular Visual Odometry, arXiv:1611.06069, published 18 Nov. 2016, discloses such a system. However, there are various problems with known systems that use neural networks. They tend to require training for the specific environments in which they will be used, so cannot be used in new environments without suitable training occurring first.

The present invention seeks to mitigate the above-mentioned problems. Alternatively and/or additionally, the present invention seeks to provide improved methods of determining the location of a mobile device.

SUMMARY OF THE INVENTION

In accordance with a first aspect of the invention there is provided a computer-implemented method of determining the location of a mobile device comprising a camera, comprising the steps of:

capturing, using the camera, a sequence of images over a period of time;

for pairs of consecutive images from the sequence of images, determining, using a first neural network, features indicative of the motion of the device between the time the first image of the pair of images was captured and the time the second image of the pair of images was captured;

for a sequence of consecutive images, determining, using a second neural network, features indicative of the location of the device from the features determined by the first neural network; and

for a sequence of consecutive images, determining the location of the device from the features determined by the second neural network.

By using a combination of two neural networks, it has been found that much more robust and reliable location determination is possible. In particular, the first neural network can be trained to most effectively determine features from the images that are indicative of the motion suggested by the differences between the images, which depends only on the two images and is not dependent upon historical information such as previously determined locations. However, the second neural network can at the same time be trained to most effectively determine the location of the mobile device from the features determined by the first neural network, for which historical information such as previously determined locations is extremely useful. By splitting the processing across two neural networks in this way, training for both step-wise motion and overall location can be effectively achieved. Further, both neural networks can be trained simultaneously by training the system as a whole, and so in particular the first neural network can be trained to determine whatever motion features are best for the operation of the system as a whole, rather than being trained to determine motion features with pre-selected attributes which may not in fact be the optimal type of features to use.

Preferably, the orientation of the device is determined in addition to the location of the device. Thus, the “pose” of the device is determined.

Preferably, the images of the sequence of images are monocular images.

Advantageously, the first neural network is a convolutional neural network. This type of neural network is particularly suited for operating upon data with a large number of parameters such as image data.

Advantageously, the second neural network is a recurrent neural network. In this case, preferably the second neural network is a Long Short-Term Memory neural network. Recurrent neural networks, in particular of Long Short-Term Memory type, are particular suitable for operating upon time-dependent data.

Preferably, the method further comprises the step of determining, from the features determined by the second neural network, relative location and orientation information for the device for each pair of consecutive captured images. In this case, preferably the step of determining the location of the device comprises composing the relative location and orientation information determined from the features determined by the second neural network. In other words, the location of the device is determined odometrically from successive motion estimations.

Advantageously, the method further comprising the step of determining corresponding uncertainty information for the relative location and orientation information for each pair of consecutive captured images. The uncertainty information can be used along with pose information as input to simultaneous localisation and mapping (SLAM) algorithms.

Each image of the sequence of images may have associated with its corresponding location information, and the method may further comprise the step of training the first and second neural networks using the corresponding location information. Preferably, each image also has associated with it orientation information.

The device may be an autonomous robot. The device may alternatively be a mobile telephone, wearable device or any other suitable mobile device.

In accordance with a second aspect of the invention there is provided a mobile device comprising:

memory;

a processor;

a camera;

wherein the device is arranged to:

capture, using the camera, a sequence of images over a period of time;

for pairs of consecutive images from the sequence of images, determine, using a first neural network provided by the processor, features indicative of the motion of the device between the time the first image of the pair of images was captured and the time the second image of the pair of images was captured;

for a sequence of consecutive images, determine, using a second neural network provided by the processor, features indicative of the location from the features determined by the first neural network; and for a sequence of consecutive images, determine the location of the device from the features determined by the second neural network.

Preferably, the device is arranged to determine the orientation of the device in addition to the location of the device.

Preferably, the images of the sequence of images are monocular images.

Advantageously, the first neural network is a convolutional neural network.

Advantageously, the second neural network is a recurrent neural network. In this case, preferably the second neural network is a Long Short-Term Memory neural network.

Preferably, the device is further arranged to determine, from the features determined by the second neural network, relative location and orientation information for the device for each pair of consecutive captured images. In this case, preferably the device is arranged to determine the location of the device by composing the relative location and orientation information determined from the features determined by the second neural network.

Advantageously, the device is further arranged to determine corresponding uncertainty information for the relative location and orientation information for each pair of consecutive captured images.

In accordance with a third aspect of the invention there is provided a computer program product arranged, when executed on a mobile device, to perform any of the methods described above.

In accordance with a fourth aspect of the invention there is provided a computer program product arranged, when executed on a mobile device, to provide any of the mobile devices described above.

It will of course be appreciated that features described in relation to one aspect of the present invention may be incorporated into other aspects of the present invention. For example, the method of the invention may incorporate any of the features described with reference to the mobile device of the invention and vice versa.

