BACKGROUND

Due to the use of mobile devices, such as smartphones and tablets, user interaction with mobile applications has been increasing. But unlike web-based applications, mobile applications can differ significantly in the features they provide. For example, link structure, the user interface, and interaction with other applications can be inconsistent from one app to another. Additionally, because mobile applications are conventionally closed (e.g., cannot be crawled), the actions of the user cannot be used as context to improve the user experience, such as personalizing search, targeting advertising, and providing personalized suggestions and assistance.

SUMMARY

Implementations capture an image of a screen on a mobile device at intervals and analyze the screen content via recognition tools to provide context for improving the user experience. For example, in some implementations, the system performs entity detection in a mobile app environment. To provide context for disambiguation, the system may group some of the captured images into a window. The window may represent a fixed length of time, with some portions of the window providing context for entities occurring in the other portions. In some implementations, the system is adaptive so the window is larger when the user's screen is static (e.g. no scrolling). Entities may be disambiguated, ranked, and associated with a user profile. In some implementations, the system may generate annotation data to provide personalized assistance the user. The annotation data may provide a visual cue for actionable content, entities or content relevant to the user, summary information, etc. The annotation data may present the annotation content, and also provide additional content, such as labels, image labels, expunge areas, etc. In some implementations, the system may index the captured images, for example by text and/or entities identified from an image. The system may use the index in various ways, such as allowing a user to search for previously viewed content, to provide context-based assistance, and to automate user input. In some implementations, the system enables the user to share a current screen or previously captured screens with another user. In some implementations, the system may track or capture user input actions, such as taps, swipes, text input, or any other action the user takes to interact with the mobile device and use this information to learn and automate actions to assist the user. In some implementations, the system may use additional data, such as the location of the mobile device, ambient light, device motion, etc. to enhance the analysis of screen data and generation of annotation data.

In one aspect, a mobile device includes at least one processor and memory storing instructions that, when executed by the at least one processor, cause the mobile device to generate a user interface configured to capture an image of a screen displayed on the mobile device in response to a command to share the screen, receive user instructions for redacting a portion of the image, and transmit the image with the selected portion redacted pursuant to said instructions to a recipient device selected by the user.

In another aspect, a method includes receiving an identifier for a recipient and an image from a first mobile device, the image representing a captured screen of the first mobile device, copying the image to an image repository associated with the recipient, and performing recognition on the image. The method may also include generating annotation data for the image based on the recognition, the annotation data including at least one visual cue; and providing the image and the annotation data to a second mobile device, the second mobile device being associated with the recipient.

In another aspect, a method includes receiving, at a second mobile device, an image of a screen captured from a display of a first mobile device, and displaying annotation data generated for the image with the image on a display of the second mobile device, the annotation data including at least one visual cue corresponding to content in the image relevant to a user of the second mobile device.

In another aspect, a method may include receiving, at a second mobile device, a plurality of screen images captured on a first mobile device, each image of the plurality of images having a corresponding timestamp, the plurality of images including a reference image with an earliest timestamp and a target image with a latest timestamp, receiving a plurality of user input actions captured on the first mobile device, each input action having a corresponding timestamp that falls between the earliest timestamp and the latest timestamp, and replaying the user input actions on the second mobile device starting from a state corresponding to the reference image.

In one general aspect, a computer program product embodied on a computer-readable storage device includes instructions that, when executed by at least one processor formed in a substrate, cause a computing device to perform any of the disclosed methods, operations, or processes. Another general aspect includes a system and/or a method for detection and ranking of entities from mobile screen content, substantially as shown in and/or described in connection with at least one of the figures, and as set forth more completely in the claims. Another general aspect includes a system and/or a method for highlighting important or user-relevant mobile onscreen content, substantially as shown in and/or described in connection with at least one of the figures, and as set forth more completely in the claims. Another general aspect includes a system and/or a method for providing actions for mobile onscreen content, substantially as shown in and/or described in connection with at least one of the figures, and as set forth more completely in the claims. Another general aspect includes a system and/or a method for providing insight for mobile onscreen content, substantially as shown in and/or described in connection with at least one of the figures, and as set forth more completely in the claims. Another general aspect includes a system and/or a method for indexing mobile onscreen content, substantially as shown in and/or described in connection with at least one of the figures, and as set forth more completely in the claims. Another general aspect includes a system and/or a method for automating user input and/or providing assistance from interaction understanding, substantially as shown in and/or described in connection with at least one of the figures, and as set forth more completely in the claims. Another general aspect includes a system and/or a method for sharing mobile onscreen content, substantially as shown in and/or described in connection with at least one of the figures, and as set forth more completely in the claims.

One or more of the implementations of the subject matter described herein can be implemented so as to realize one or more of the following advantages. As one example, implementations may provide a consistent user experience across mobile applications, so that similar type of actionable content behaves the same across applications. As another example, implementations provide context for personalizing certain tasks, such as ranking search results and providing assistance. As another example, implementations provide an interface to quickly discover user-relevant and content relevant content on the screen and to surface insightful relationships between entities displayed in the content. As another example, implementations may allow a user of a mobile device to share a screen with another user or to transfer the state of one mobile device to another mobile device. Implementations may also allow a mobile device to automatically perform a task with minimal input from the user.

The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

