BACKGROUND

Users are often inundated with electronic communications such as emails, SMS communications, and social networking communications. Many electronic communications that are sent to a user explicitly solicit a reply or otherwise contain information to which the user may wish to reply. For example, an email that contains “Are you interested in grabbing lunch?” may explicitly solicit a reply. An email that contains “I just got engaged!” may not explicitly solicit a reply, but a user that receives the email may still wish to send a congratulatory reply to the email. Users formulate replies to the electronic communications and input the formulated replies at computing devices utilizing a keyboard (e.g., a virtual keyboard of a smartphone or tablet), a microphone, and/or other user interface input device(s) of the computing device.

SUMMARY

This specification is directed generally to methods and apparatus related to determining reply content for a reply to an electronic communication. Some implementations are directed generally toward analyzing a corpus of electronic communications (e.g., emails, SMS communications, and/or social networking communications) to determine relationships between one or more original message features of “original” messages of electronic communications and reply content that is included in “reply” messages of those electronic communications. For example, the corpus may be analyzed to determine a relationship between the original message n-grams “attached” and “draft document(s) for your review” and the reply n-gram “I'll take a look and will provide feedback” based on the original message n-grams occurring frequently in original messages that were replied to with the reply n-gram (and optionally other content). In some implementations, the relationships between the original message feature(s) and the reply content may be determined without allowing direct human access to one or more (e.g., any) of the electronic communications of the corpus. In some implementations, the relationships between the original message feature(s) and the reply content may be determined based on training a machine learning system with training examples each having one or more output parameters indicative of reply content of a reply message of a respective electronic communication and one or more input parameters indicative of original content of the original message of the respective electronic communication.

Some implementations are directed generally toward providing reply text to include in a reply to a communication based on determined relationships between one or more message features of the communication and the reply text. For example, an email may be sent to the user and include “Our team just won the Awesomeness Award!” and the reply text “Good job!” may be provided as a candidate reply to the email based on a mapping (data defining a relationship) between one or more features of the email and “Good job!”. For instance, the reply text may be provided for auto-population in a reply to the email, provided for presentation (visually or audibly) as a suggestion for selection by the user, etc. As another example, an email may be sent to the user and include “Our team just won the Awesomeness Award!” and the reply text “Good job!” may be provided to the user as a reply to the email based on a mapping between one or more features of the email and reply content that is indicative of a congratulatory n-gram in general—and a mapping that defines “Good job!” as a highly ranked congratulatory n-gram. For instance, “Good job!” may be defined as a highly ranked congratulatory n-gram based on it being a popular congratulatory n-gram for the user, for a group of users similar to the user, or for all users. In some implementations, the reply text may be provided for inclusion in a reply to a communication independent of any textual input provided by the user in generating the reply to the communication. In other words, in some implementations the reply text may be presented to the user before the user has even started typing the reply. In some implementations, the reply text to include in a reply to a communication may be determined based on output provided by a trained machine learning system in response to providing the trained machine learning system with one or more message features of the communication.

In some implementations, a computer implemented method may be provided that is performed by at least one computer system and includes the steps of: identifying a corpus of electronic communications of a plurality of users, the electronic communications of the corpus each including a reply message and an original message to which the reply message is responsive; determining reply content that occurs in the reply message of a plurality of the electronic communications of the corpus; determining at least one message feature based on the original message of a plurality of the electronic communications that include the reply content in the reply message; calculating, based at least in part on a quantity of the electronic communications that include both the reply content and the at least one message feature, a strength of relationship value between the reply content and the at least one message feature; and defining a relationship between the at least one message feature and the reply content based on the strength of relationship value, the relationship defined for use in electronically providing the reply content for presentation to recipients of additional electronic communications that include the at least one message feature.

This method and other implementations of technology disclosed herein may each optionally include one or more of the following features.

In some implementations, defining the relationship between the at least one message feature and the reply content based on the strength of relationship value comprises: defining the relationship; and assigning the strength of relationship value to the relationship.

In some implementations, defining the relationship between the at least one message feature and the reply content based on the strength of relationship value comprises: determining the strength of relationship value satisfies a threshold; and defining the relationship based on determining the strength of relationship value satisfies the threshold.

In some implementations, the method further comprises: identifying an electronic communication sent to a user; determining the electronic communication includes the at least one message feature; identifying the reply content based on the relationship between the at least one message feature and the reply content; and providing electronically the reply content for potential inclusion in a reply to the electronic communication. In some of those implementations, identifying and providing the reply content for potential inclusion in the reply to the electronic communication occurs independent of any textual input provided via a computing device of the user in generating the reply to the message. In some of those implementations, the strength of relationship value is assigned to the relationship between the at least one message feature and the reply content, and the method further comprises: identifying additional reply content based on an additional relationship between the at least one message feature and the additional reply content; and identifying an additional strength of relationship value for the additional relationship; wherein providing the reply content for potential inclusion in the reply to the electronic communication is based on comparison of the strength of relationship value and the additional strength of relationship value. Providing the reply content for potential inclusion in the reply to the electronic communication based on comparison of the strength of relationship value and the additional strength of relationship value may comprise: providing the reply content for potential inclusion in the reply to the electronic communication without providing the additional reply content for potential inclusion in the reply to the electronic communication; or determining a display prominence for each of the reply content and the additional reply content based on the strength of relationship value and the additional strength of relationship value, and providing both the reply content and the additional reply content (with an indication of the display prominences) for potential inclusion in the reply to the electronic communication.

In some implementations, the at least one message feature includes an n-gram in text of the original message of the electronic communications. In some of those implementations, determining the at least one message feature comprises determining the n-gram based on at least one of: a count of occurrences of the n-gram in the electronic communications that include the reply content in the reply message; and an overall popularity measure associated with the n-gram.

In some implementations, determining the reply content that occurs in the reply message of the plurality of the electronic communications of the corpus comprises: grouping the electronic communications of the corpus into a plurality of clusters based on similarities between text in the reply messages of the electronic communications of the corpus; and determining the reply content based on the text of the reply messages of one cluster of the clusters. In some of those implementations, the reply content is an indicator of a plurality of unique n-grams that have similar semantic meanings. In some of those implementations, the method further comprises assigning a plurality of n-grams to the reply content, the n-grams being determined based on a quantity of the reply messages of the one cluster that include the n-grams. In some of those implementations, determining the at least one message feature based on the original message of the plurality of the electronic communications that include the reply content comprises determining the at least one message feature based on text of the original messages of the one cluster; and calculating the strength of relationship value between the reply content and the at least one message feature is based at least in part on a quantity of the electronic communications of the one cluster that include both the reply content and the at least one message feature.

In some implementations, the method further comprises: annotating text of the original message of one or more of the electronic communications with one or more annotations; wherein the at least one message feature includes an n-gram that includes one or more terms from the text of the one or more electronic communications and includes one or more of the annotations. In some of those implementations, the at least one message feature includes the one or more terms from the text of the one or more electronic communications and the one or more of the annotations includes an entity category that replaces one or more other terms of the original message of the one or more electronic communications.

In some implementations, identifying the corpus of communications comprises: selecting the electronic communications for inclusion in the corpus based on the reply text of each electronic communications being less than a reply text threshold length.

In some implementations, the method further comprises: generating training examples each having an output parameter indicative of the reply content and one or more input parameters indicative of the at least one message feature of a respective of the electronic communications that include the reply content; and training a machine learning system based on the training examples, the training the machine learning system including calculating the strength of relationship value and defining the relationship. In some of those implementations, determining the reply content that occurs in the reply message of the plurality of the electronic communications of the corpus comprises: grouping the electronic communications of the corpus into a plurality of clusters based on similarities between text in reply messages of the electronic communications of the corpus; and determining the reply content based on the text of the reply message of electronic communications of one cluster of the clusters.

In some implementations, a computer implemented method may be provided that is performed by at least one computer system and includes the steps of: identifying an electronic communication sent to a user; determining at least one message feature based on the electronic communication; determining, independent of any textual input provided via a computing device of the user in generating a reply to the electronic communication, candidate reply text for inclusion in the reply to the message, the determining based on at least one defined relationship between the at least one message feature and the candidate reply text; and providing the candidate reply text for inclusion in the reply to the electronic communication.

This method and other implementations of technology disclosed herein may each optionally include one or more of the following features.

In some implementations, determining the candidate reply text comprises determining a strength of relationship value between the at least one message feature and the candidate reply text; and providing the candidate reply text for inclusion in the reply to the message is based on the strength of relationship value.

In some implementations, determining the candidate reply text comprises: identifying a first relationship of the at least one defined relationship, the first relationship being between a first message feature, of the at least one message feature, and the candidate reply text; identifying a second relationship of the at least one defined relationship, the second relationship being between a second message feature, of the at least one message feature, and the candidate reply text; and identifying a first strength of relationship value for the first relationship and a second strength of relationship value for the second relationship; and providing the candidate reply text for inclusion in the reply to the message is based on the first strength of relationship value and the second strength of relationship value.

In some implementations, a strength of relationship value is assigned to the relationship between the at least one message feature and the candidate reply text, and the method further comprises: identifying an additional candidate reply text based on an additional relationship between the at least one message feature and the additional reply content; identifying an additional strength of relationship value for the additional relationship; and providing the reply content for potential inclusion in the reply to the message is based on comparison of the strength of relationship value and the additional strength of relationship value. In some of those implementations, providing the candidate reply text for potential inclusion in the reply to the message based on comparison of the strength of relationship value and the additional strength of relationship value comprises: providing the candidate reply text for potential inclusion in the reply to the message without providing the additional candidate reply text for potential inclusion in the reply to the message. In some of those implementations, providing the candidate reply text for potential inclusion in the reply to the message based on comparison of the strength of relationship value and the additional strength of relationship value comprises: determining a display prominence for each of the candidate reply text and the additional reply content based on the strength of relationship value and the additional strength of relationship value; and providing both the candidate reply text and the additional candidate reply text for potential inclusion in the reply to the message, wherein the candidate reply text and the additional candidate reply text are provided with an indication of the display prominences.

In some implementations, the at least one message feature includes an n-gram in the original text of the one or more electronic communications.

In some implementations, determining the candidate reply text comprises: identifying a first relationship of the at least one defined relationship, the first relationship being between the at least one message feature and a category of reply; identifying a second relationship of the at least one defined relationship, the second relationship being between the category of reply and the reply text. In some of those implementations, the method further comprises: determining a second strength of relationship value for the second relationship based at least in part on content of the electronic communication and/or a quantity of past replies of the user that include the reply text; and providing the candidate reply text for inclusion in the reply to the message is based on the second strength of relationship value.

In some implementations, determining the candidate reply text comprises: providing the at least one message feature as input to a trained machine learning system and receiving an indication of the candidate reply text as output from the trained machine learning system. In some of those implementations, the indication of the candidate reply text is a category of reply and the reply text is determined based on a defined relationship between the reply text and the category of reply.

Other implementations may include a non-transitory computer readable storage medium storing instructions executable by a processor to perform a method such as one or more of the methods described above. Yet another implementation may include a system including memory and one or more processors operable to execute instructions, stored in the memory, to perform a method such as one or more of the methods described above.

It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

