BACKGROUND

Many mobile devices, such as smartphones, come equipped with a camera for capturing pictures and digital images. Mobile devices provide access to a wide variety of information. This information may be viewed in web browsers or native applications running on a mobile device. Users can also use the images to obtain additional information related to an object or location. For example, digital image files, video and/or audio files, as well as resources for particular subjects are accessible over the Internet.

Object recognition software is used to detect and identify objects in an image or video sequence. Processes for object recognition of an image or video sequence often employ various distinct steps for analyzing an image to positively identify an object within the image.

Once a user has selected an image in which to look for objects of interest, the object recognition software running in the background of a mobile device and/or on a server will operate without user feedback during the process of identifying objects within the image. Oftentimes a lack of user feedback during the object recognition software leads to imperfect object recognition including incorrect identification, no positive identification, or an undesired object being identified within the image or video sequence. However, without direct visual indicators of the process of object recognition displayed within the candidate image on a mobile device, the user can have little understanding of the object recognition process and cannot give important feedback to assist in the object recognition process.

SUMMARY

This specification describes technologies relating to representing an object recognition process in real-time using user interface elements to a user viewing the image on a mobile device.

In general, one innovative aspect of the subject matter described in this specification can be embodied in computer-implemented methods that include capturing, by a camera on a user device, a plurality of successive images for display in an application environment of an application executing on the user device, performing an object recognition process on the images, the object recognition process including determining that a plurality of images, each depicting a particular object, are required to perform object recognition on the particular object, and in response to the determination, generating, within the application environment, a user interface element that indicates a camera operation to be performed, the camera option capturing two or more images, determining that a user, in response to the user interface element, has caused the indicated camera operation to be performed to capture the two or more images, and in response, determining whether a particular object is positively identified from the plurality of images, and in response to determining that the particular object is positively identified, eliminating the user interface element that indicates the camera operation to be performed and generating a user interface element that indicates the positive identification of the particular object, and in response to determining that the particular object is not positively identified, maintaining the user interface element that indicates the camera operation to be performed. Other embodiments of this aspect include corresponding apparatus, methods, systems, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

These and other aspects can each optionally include one or more of the following features. In some aspects, the user interface element includes a first interface element that indicates capturing images of the particular object from multiple angular positions relative to a current angular position of the camera to the particular object. In some aspects, the user interface element includes a second user interface element that is different from the first user interface element and that indicates capturing an image of the particular object from a particular angular position relative to a current angular position of the camera to the particular object.

In some aspects, the user interface element indicates capturing an image of the particular object at a zoom level that is different from a current zoom level of the camera.

In some aspects, the step of determining whether a particular object is positively identified from the plurality of images includes sending, by the application executing on the user device, the collected plurality of images to an object recognition server, and receiving, at the user device from the object recognition server, an indication of whether the particular object is positively identified. In some aspects, sending the collected plurality of images to the object recognition server includes determining, based on a bandwidth criteria, a number of images of the plurality of images that depict the particular object from a particular angle, and sending the number of images to the object recognition server.

In general, another innovative aspect of the subject matter described in this specification can be embodied in computer-implemented methods that include receiving, by an image processing component, an image depicting an object obtained from a camera device, determining that a particular object is not positively identified from the image, and in response providing data that causes an application environment of an application executing on a user device in which is displayed successive images captured by the camera on the user device to display a user interface element that indicates a camera operation to be performed, receiving, in response to the camera operation being performed, a plurality of images captured by the camera operation performed, determining whether a particular object is positively identified from the plurality of images, and in response to determining that the particular object is positively identified, providing data that causes the user interface element that indicates the camera operation to be performed to be eliminated and that causes a user interface element that indicates the positive identification of the particular object to be displayed in the application environment, and in response to determining that the particular object is not positively identified, providing data that causes the user interface element that indicates the camera operation to be performed to be maintained within the application environment. Other aspects of this aspect include corresponding apparatus, methods, systems, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

These and other aspects can each optionally include one or more of the following features. In some aspects, determining whether the particular object is positively identified from the plurality of images includes iteratively updating a weighted average until a weighted average threshold is exceeded. In some aspects, each iteration includes performing object recognition on each image to determine if the particular object is recognized as one or more identified objects, where for each particular object that is recognized as one identified object for a particular image, assigning a weight for the one identified object for the particular image, for each particular object that is recognized as two or more identified objects for a particular image, assigning a distributed equal weight to each of the two or more identified objects for the particular image, determining if the weighted average exceeds the weighted average threshold.

In some aspects, determining whether the particular object is positively identified from the plurality of images includes creating, from the plurality of images, a composite image, and performing object recognition on the composite image.

In some aspects, determining whether the particular object is positively identified from the plurality of images includes creating, from the plurality of images, a panoramic image, and performing object recognition on the panoramic image.

In some aspects, providing data that causes an application environment of an application executing on the user device in which is displayed successive images captured by a camera on the user device to display the user interface element that indicates a camera operation to be performed includes determining one of a plurality of camera operations to be performed, wherein each camera operation has a corresponding user interface element that indicates the camera operation and that is different from each user interface element for each other camera operation, and providing to the user device data that causes the user interface element for the determined one of the plurality of camera operations to be displayed.

In general, another innovative aspect of the subject matter described in this specification can be embodied in computer-implemented methods that include providing images captured by a camera on a user device, sending, at a first frequency and from the user device, the images to an object recognition server, receiving, at the user device and from the object recognition server, an indication that a particular object within an image is not positively identified from the images sent to the server, in response to receiving the indication that the particular object is not positively identified, sending, at a second frequency higher than the first frequency, the images to the object recognition server, receiving, at the user device from the object recognition server, an indication that the particular object is positively identified, and in response to receiving the indication that the particular object is positively identified, ceasing to send images from the user device to the object recognition server at the second frequency. Other aspects of this aspect include corresponding apparatus, methods, systems, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

These and other aspects can each optionally include one or more of the following features. In some aspects, the method can further include in response to receiving the indication that the particular object is not positively identified, generating a user interface element indicating a camera operation to be invoked by the user during an object recognition process, wherein the camera operation involves capturing two or more images, and wherein the step of sending images at the second frequency to the object recognition server is conditional upon performance of the camera operation. In some aspects, the method can further include in response to receiving the indication that the particular object is positively identified, eliminating the user interface element that indicates the camera operation to be performed and generating a user interface element that indicates the positive identification of the particular object.

Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. Providing real-time user interface elements, or visual indicators, representing an object recognition process to a user on a mobile device displaying an image of interest allows the user to provide feedback and assist in the object recognition process. Presenting user interface elements for intermediary steps in the object recognition process to a user can help to identify potential problems in the object recognition process during one or more of the steps. Moreover, should the object recognition process fail to identify an object, the use of user interface elements can help the user understand at which point the object recognition process is failing. This reduces or eliminates the need for instrumenting an object recognition process to provide, for example, data logging describing pass events and fail events during the process. Accordingly, the technological fields of object recognition processing and process troubleshooting are improved.

Moreover, providing visual indicators about acquiring multiple angles of an object may assist a user in selecting a second image if a first image is determined to be unsuitable for the object recognition process. The visual indicators selected to represent the steps of an object recognition process may be built around a human-computer interaction model to provide an improved intuitive user experience. Moreover, introducing latency into the object recognition process may allow a user to more easily visually understand a process that otherwise may occur too quickly for user comprehension.

In accordance with some aspects described herein, bandwidth usage may be controlled by directing the user through the user interface elements to take key image frames in the user interface (e.g. different angles), and only sending the server the selected key frames to be analyzed for object recognition. This can result in more efficient utilization of bandwidth and computing resources.

With conventional user interfaces, a user may not know to move the camera of the user device to different angles for a candidate object in order to perform object recognition. An effective user interface would ideally enable the user to readily and rapidly access the right data/functionality. The object recognition application of the present system can present visual indications for a user to obtain images of an object from different angles. For example, the user can be viewing on the mobile device an object of interest using the object recognition application, but the current angle or view of the object is unable to recognize the object with the current image data. The object recognition application, through a user interface element generator, presents user interface elements (e.g., virtual indicators, arrows, etc.) to the user to obtain images of the object from another angle. Thus, the present system enables the user to readily and rapidly access the angle of the object with ease and speed of navigation, particularly on small screen devices.

The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

