TECHNICAL FIELD

This application is directed to the field of transmission of information, and more particularly to the field of preparing digital data for mobile communications.

BACKGROUND

Mobile chat has become a mainstream worldwide communication medium. According to market data, message volume for mobile chat has long exceeded a volume of mobile operator supported SMS (Short Messaging Service) communications: for example, about 19 billion mobile chat messages have been sent daily over mobile chat applications in 2012, compared with 17.5 billion SMS messages. A disparity between mobile chat applications and traditional operator-supported messaging is expected to quickly grow; thus, volume forecasts for 2014 by market research companies are in the range of 50 billion mobile chat messages vs. 21 billion SMS messages per day. Some experts point out that the growth numbers for mobile chat may be much higher if an acceptance of an industry wide Joyn platform for unified messaging and file sharing across mobile platforms and operators quickly progresses.

A cumulative number of users for the top eight mobile chat applications has exceeded 1.5 billion people in 2013. These applications are: WhatsApp (recently acquired by Facebook, Inc.), BlackBerry Messenger, Viber, Nimbuzz, Apple's iMessage, KakaoTalk, Facebook Messenger for Android, and TenCent QQ and WeChat. Mobile operators and smartphone manufacturers are starting to add dedicated buttons to launch popular mobile chat applications with certain models of their phones, such as a WhatsApp button on a Nokia Asha 210 phone.

Wearable devices of different types, including Head-Mounted Displays (HMDs) and cameras, such as Google Glass, are expected to increasingly interoperate with mobile phones and applications, which may add new dimensions to mobile chat applications and pair them with augmented reality and sensor data.

One key advantage of mobile chat applications that makes them a desirable communications medium for a mass user is associated with their multimedia sharing and platform integration capabilities. For example, each of the three leading mobile chat applications in Asia: WeChat in China, KakaoTalk in Korea, and LINE in Japan are currently offering emoji, stickers and mobile games directly integrated into chat platforms; LINE has launched a line of cartoon characters with their own show and toys; KakaoTalk and Evernote Corporation of Redwood City, Calif. have announced integration of Evernote service features into the KakaoTalk chat platform. File and screen sharing and other enhanced communications features are already present in popular chat applications like Skype and have been included into short-term product roadmaps for other mobile chat applications.

Notwithstanding significant progress in the proliferation of mobile chat applications and in the development of features of mobile chat applications, mobile messaging still suffers from significant shortcomings with respect to capabilities of individual applications and the role of the applications within an overall mobile software ecosystem. One of the most noticeable flaws of mobile chat applications is an absence of meaningful integration of such applications into mainstream mobile software. Thus, in a widespread usage scenario, a person needs to communicate with other chat participants in conjunction with current on-screen and application content: images, maps, documents, web pages, charts, video, etc. However, the use of mobile chat applications is predominantly disconnected from such mainstream content consumption; it takes a significant effort and many steps to explain to other chat participants the nature of a user inquiry or a statement related to the present content on that user's screen. Additionally, existing methods for sharing such content are generally available only at the file exchange level and are time consuming and awkward.

Similarly, in an augmented reality context, a user may function within real life scenes where a communication need may be invited by a current environment with its changing features and situations. The existing mobile chat applications are poorly suited to meet such real-time requirements.

Accordingly, it is desirable to develop a cohesive interoperation between mainstream mobile applications and content management systems, on the one hand, and mobile chat applications, on the other hand.

SUMMARY

According to the system described herein, integrating an application and a chat session includes a first user identifying content from the application, wherein the application is independent of any chat session, the first user initiating the chat session with at least a second user that is remote from the first user by transmitting the content from the first user to the second user, where the first and second user establish the chat session in response to the content being transmitted, and the first user and the second user sharing the content in connection with the chat session. The content may be a portion of a photograph, a portion of an image, a portion of text from a document, and/or a portion of a screen. The portion may be automatically identified following the user pointing and marking up the content. The first user may mark up the content before transmitting it to the second user. The first user may initiate the chat with the content by a multi-touch gesture, a keyboard shortcut, and/or pointing an arrow drawn by software on a screen of a device containing the content. One of the users may facilitate the chat session using an augmented reality system and the content may be part of a real scene viewed by the one of the users with the augmented reality system. The second user may be part of a list of participants defined by the first user and/or part of a list of participants engaging in recent chat sessions with the first user. The second user may modify and mark up the content. The second user may be prohibited from marking up the content and/or modifying the content. Prior to transmission of the content, the first user may select from a set of standard comments/questions that are transmitted to the second user with the content. The first user may confirm the content prior to transmission. The chat session may run simultaneously on multiple devices of each of the users. The application may run on a non-mobile device of one of the users and the chat session may run simultaneously on a mobile device of the one of the users. Content may be selected on the non-mobile device. The chat session may be provided on a device using WhatsApp, BlackBerry Messenger, Viber, Nimbuzz, iMessage, KakaoTalk, Facebook Messenger, TenCent QQ, WeChat, or Skype. The chat session may be provided on a device using a browser. The chat session may be a mobile chat session. The content may be provided on at least one mobile device.

According further to the system described herein, computer software, provided in a non-transitory computer-readable medium, integrates an application and a chat session. The software includes executable code that identifies, for a first user, content from the application, where the application is independent of any chat session, executable code that initiates, for the first user, the chat session with at least a second user that is remote from the first user by transmitting the content from the first user to the second user, where the first and second user establish the chat session in response to the content being transmitted, and executable code that shares the content between the first user and the second user in connection with the chat session. The content may be a portion of a photograph, a portion of an image, a portion of text from a document, and/or a portion of a screen. The portion may be automatically identified following the user pointing and marking up the content. The content may be marked up before being transmitted it to the second user. The chat may be initiated using the content by a multi-touch gesture, a keyboard shortcut, and/or pointing an arrow drawn by software on a screen of a device containing the content. An augmented reality system may be used and the content may be part of a real scene viewed by one of the users with the augmented reality system. The second user may be part of a list of participants defined by the first user and/or part of a list of participants engaging in recent chat sessions with the first user. The second user may modify and mark up the content. The second user may be prohibited from marking up the content and/or modifying the content. Prior to transmission of the content, the first user may select from a set of standard comments/questions that are transmitted to the second user with the content. The first user may confirm the content prior to transmission. The chat session may run simultaneously on multiple devices of each of the users. The application may run on a non-mobile device of one of the users and the chat session may run simultaneously on a mobile device of the one of the users. Content may be selected on the non-mobile device. The chat session may be provided on a device using one of: WhatsApp, BlackBerry Messenger, Viber, Nimbuzz, iMessage, KakaoTalk, Facebook Messenger, TenCent QQ, WeChat, or Skype. The chat session may be provided on a device using a browser. The chat session may be a mobile chat session. The content may be provided on at least one mobile device.

The proposed system integrates chat applications into mobile and desktop software workflow, including methods for selecting portions of content for transmission to chat participants, methods for identifying user instructions to initiate chat session and transmit the selected content, including pointing to the content, and sharing of on-screen content with chat participants with a possible addition of user authored or standard comments.

Integration of chat systems into software may include an ability to extract (clip) portions of on-screen content displayed to a user in desktop and mobile applications such as web browsers, document viewing and authoring applications (for example, office applications, image and video viewers and editors, photo capturing and editing software, games, etc.), and instantly initiate a mobile chat session where the extracted content is transmitted and shared between chat participants. This approach cardinally differs from a non-transparent content transmission method in the existing mobile chat applications, as explained elsewhere herein.

Depending on a platform, device, software title and interoperability features, such content extraction may represent clipped content in different formats: it may be a snapshot of a selected portion of a user screen or a whole screen in a common image format, with or without added user markup and/or caption; a portion of plain or formatted text of a document; a photo taken in an original photo capturing or image editing application, with or without subsequent user markup; etc. Content may also be captured and transmitted in several alternative formats and then displayed to chat participants in one of the available formats, depending on participants' chat settings and capabilities of their client (browser based chat software).

Content selection may be done via native software selection means, such as contiguous selection of textual fragments or a range of table cells in office software; visual selection using graphical capabilities utilized in image editors or in markup software, such as Evernote Skitch; etc. Additionally, selection software within an integrated chat system may be able to automatically choose a portion of content to which a user has pointed as a natural selection candidate.

A chat session with a selected content may be initiated via a dedicated gesture or a shortcut, such as pointing an arrow to a selected fragment, making a multi-touch gesture, entering a keyboard shortcut, etc. Such user instruction may result in an immediate opening of native mobile chat software or a browser based version of mobile chat software, starting a new chat session if the user is not in a session, and transmitting the selected content to chat participants. The list of participants may be defined on the fly via explicit user choice from the list of contacts or may use various other options, such as indicating participants of a most recent chat session as a default option or suggesting most frequent participants of chat sessions with the user as a priority list. Specific display formats for the transmitted content may be different for different participants of the chat session, as explained elsewhere herein. For some of the participants, the transmitted content may be represented in an editable format, allowing the participants to modify the original content and share the modified fragment. For other participants, markup on top of the received content without modifying the transmitted fragment may be available in chat client software used by the other participants. Yet another group of participants may only be able to add comments to the received fragment without any editing or markup features.

Prior to transmitting a selected fragment of content, the system may display to sender a standard set of comments or questions to be transmitted with the content, such as “what is this?”, “how do I get there?”, “which one?”, “look how great it is!”, etc. A chosen comment or question may be added to the transmitted content, possibly with additional textual comments explicitly added by the sender. Users may be able to customize and augment lists of standard comments and questions.

When a user points to an object to initiate a chat session without an explicit content selection, the system may determine a selected candidate and either highlight the selected candidate for approval by the user prior to transmission or immediately transmit the selected candidate to chat participants. For example, a user may point a session initialization arrow to a textual content and the system may offer in response a paragraph of text to which the user is pointing as a natural selection candidate. In some embodiments, there may be additional conventions regarding disposition of a pointing arrow with respect to text causing selection of different portions of text (in the absence of an explicit content selection by the user). For example, pointing an arrow nearly vertically to a line of text may be interpreted as a line selection, while pointing the same arrow nearly horizontally to a middle of a paragraph of text without touching the text may cause a selection of the whole paragraph. Other examples may include images, drawings, charts or other objects within a document, pointing to which may cause suggesting of a target object as a selection candidate.

The system may run on multiple devices of a single user. For example, the user may be viewing or editing content on a tablet, a notebook or a desktop computer and may initiate a chat session from a smartphone by pointing to a portion of content on the tablet, notebook or desktop. In this case, the content may be first transmitted to the smartphone of the user using any available network connection, such as Bluetooth, Wi-Fi, WAN (possibly using a cloud service to temporarily store and transmit content) or any combination of transmission mechanisms.

For augmented reality systems, such as Google Glass, the system presents an opportunity to instantly point out an object or area of interest on a photograph of a real scene the user is looking at, and transmit the object or area of interest to other chat participants for answers, comments and markup by the other participants, which may direct further actions of the user.

