<jupyter_start><jupyter_text><jupyter_code>import IPython.display
IPython.display.YouTubeVideo('4m44aPkLY2k')<jupyter_output><empty_output><jupyter_text>如何使用和开发微信聊天机器人的系列教程 A workshop to develop & use an intelligent and interactive chat-bot in WeChat  WeChat is a popular social media app, which has more than 800 million monthly active users.by: <NAME> (Sam)October 2018 : Update to support Python 3 in local machine, e.g. iss-vm.April 2017 ======= Scan the QR code to become trainer's friend in WeChat =====>>  第四课：自然语言处理：语义和情感分析 Lesson 4: Natural Language Processing 2* 整篇消息文字的情感分析 (Analyzing Sentiment: perform sentiment analysis)* 消息文字中语句的情感分析 (Analyzing Entity Sentiment: perform entity sentiment analysis)* 消息文字中名称实体的识别 (Analyzing Entities: perform entity analysis)* 语句的语法分析 (Analyzing Syntax: syntactic analysis)* 消息文字的主题分类 (Classifying Content: analyze content to determine the content category)  Using Google Cloud Platform's Machine Learning APIs From the same API console, choose "Dashboard" on the left-hand menu and "Enable API".Enable the following APIs for your project (search for them) if they are not already enabled:** Google Cloud Natural Language API **Finally, because we are calling the APIs from Python (clients in many other languages are available), let's install the Python package (it's not installed by default on Datalab) ---  Install the client library for Language Processing<jupyter_code># !pip install --upgrade google-cloud-language<jupyter_output><empty_output><jupyter_text>---<jupyter_code># Imports the Google Cloud client library
from google.cloud import language
from google.cloud.language import enums
from google.cloud.language import types
# Imports useful library
import six, sys
# [Optional] Display location of service account API key if defined in GOOGLE_APPLICATION_CREDENTIALS
!echo $GOOGLE_APPLICATION_CREDENTIALS
    
##################################################################
# (1) Instantiates a client - using GOOGLE_APPLICATION_CREDENTIALS
#     client_language = language.LanguageServiceClient()
# 
# (2) Instantiates a client - using 'service account json' file
    client_language = language.LanguageServiceClient.from_service_account_json(
        "/media/sf_vm_shared_folder/000-cloud-api-key/mtech-ai-7b7e049cf5f6.json")
##################################################################<jupyter_output><empty_output><jupyter_text>* 整篇消息文字的情感分析 (Analyzing Sentiment: perform sentiment analysis)  testimony example<jupyter_code>def didi_sentiment_text(text):    
    # Imports the Google Cloud client library
    from google.cloud import language
    from google.cloud.language import enums
    from google.cloud.language import types
    
##################################################################
# (1) Instantiates a client - using GOOGLE_APPLICATION_CREDENTIALS
#     client_language = language.LanguageServiceClient()
# 
# (2) Instantiates a client - using 'service account json' file
    client_language = language.LanguageServiceClient.from_service_account_json(
        "/media/sf_vm_shared_folder/000-cloud-api-key/mtech-ai-7b7e049cf5f6.json")
##################################################################

    document = types.Document(
        content=text,
        type=enums.Document.Type.PLAIN_TEXT)

    # Detects the sentiment of the text
    sentiment = client_language.analyze_sentiment(document=document).document_sentiment

    print('Text: {}'.format(text))
    print(' ')
    print('Sentiment: {}, {}'.format(sentiment.score, sentiment.magnitude))
    
    return sentiment
text=u'<NAME>: One of the most useful things we learnt at ISS was to be analytical and to be able to adapt, \
       especially in an industry that is changing so quickly. What ISS gave us was an invaluable reference point.'
cloud_response = didi_sentiment_text(text)
print('[ cloud_response ]\n{}'.format(cloud_response))<jupyter_output>[ cloud_response ]
magnitude: 1.899999976158142
score: 0.8999999761581421<jupyter_text>* 消息文字中语句的情感分析 (Analyzing Entity Sentiment: perform entity sentiment analysis)<jupyter_code>def didi_entity_sentiment_text(text):
    """Detects entity sentiment in the provided text."""
##################################################################
# (1) Instantiates a client - using GOOGLE_APPLICATION_CREDENTIALS
#     client_language = language.LanguageServiceClient()
# 
# (2) Instantiates a client - using 'service account json' file
    client_language = language.LanguageServiceClient.from_service_account_json(
        "/media/sf_vm_shared_folder/000-cloud-api-key/mtech-ai-7b7e049cf5f6.json")
##################################################################

    if isinstance(text, six.binary_type):
        text = text.decode('utf-8')

    document = types.Document(
        content=text.encode('utf-8'),
        type=enums.Document.Type.PLAIN_TEXT)

    # Detect and send native Python encoding to receive correct word offsets.
    encoding = enums.EncodingType.UTF32
    if sys.maxunicode == 65535:
        encoding = enums.EncodingType.UTF16

    result = client_language.analyze_entity_sentiment(document, encoding)

    for entity in result.entities:
        print('Mentions: ')
        print(u'Name: "{}"'.format(entity.name))
        for mention in entity.mentions:
            print(u'  Begin Offset : {}'.format(mention.text.begin_offset))
            print(u'  Content : {}'.format(mention.text.content))
            print(u'  Magnitude : {}'.format(mention.sentiment.magnitude))
            print(u'  Sentiment : {}'.format(mention.sentiment.score))
            print(u'  Type : {}'.format(mention.type))
        print(u'Salience: {}'.format(entity.salience))
        print(u'Sentiment: {}\n'.format(entity.sentiment))
        
    return result
text=u'<NAME>: One of the most useful things we learnt at ISS was to be analytical and to be able to adapt, \
       especially in an industry that is changing so quickly. What ISS gave us was an invaluable reference point.'
cloud_response = didi_entity_sentiment_text(text)
print('[ cloud_response ]\n{}'.format(cloud_response))<jupyter_output>[ cloud_response ]
entities {
  name: "<NAME>"
  type: PERSON
  salience: 0.6122507452964783
  mentions {
    text {
      content: "<NAME>"
    }
    type: PROPER
    sentiment {
      magnitude: 0.20000000298023224
      score: 0.20000000298023224
    }
  }
  mentions {
    text {
      content: "One"
      begin_offset: 12
    }
    type: COMMON
    sentiment {
      magnitude: 0.8999999761581421
      score: 0.8999999761581421
    }
  }
  sentiment {
    magnitude: 1.2000000476837158
    score: 0.6000000238418579
  }
}
entities {
  name: "industry"
  type: OTHER
  salience: 0.1542702317237854
  mentions {
    text {
      content: "industry"
      begin_offset: 129
    }
    type: COMMON
    sentiment {
    }
  }
  sentiment {
    magnitude: 0.699999988079071
    score: 0.30000001192092896
  }
}
entities {
  name: "things"
  type: OTHER
  salience: 0.12234412133693695
  mentions {
    text {
      content: "things"
      begin_offset: 35
    }
    type: COMMON
    sentiment[...]<jupyter_text>* 消息文字中名称实体的识别 (Analyzing Entities: perform entity analysis)<jupyter_code>def didi_entities_text(text):
    """Detects entities in the text."""
##################################################################
# (1) Instantiates a client - using GOOGLE_APPLICATION_CREDENTIALS
#     client_language = language.LanguageServiceClient()
# 
# (2) Instantiates a client - using 'service account json' file
    client_language = language.LanguageServiceClient.from_service_account_json(
        "/media/sf_vm_shared_folder/000-cloud-api-key/mtech-ai-7b7e049cf5f6.json")
##################################################################

    if isinstance(text, six.binary_type):
        text = text.decode('utf-8')

    # Instantiates a plain text document.
    document = types.Document(
        content=text,
        type=enums.Document.Type.PLAIN_TEXT)

    # Detects entities in the document. You can also analyze HTML with:
    #   document.type == enums.Document.Type.HTML
    entities = client_language.analyze_entities(document).entities

    # entity types from enums.Entity.Type
    entity_type = ('UNKNOWN', 'PERSON', 'LOCATION', 'ORGANIZATION',
                   'EVENT', 'WORK_OF_ART', 'CONSUMER_GOOD', 'OTHER')

    for entity in entities:
        print('=' * 20)
        print(u'{:<16}: {}'.format('name', entity.name))
        print(u'{:<16}: {}'.format('type', entity_type[entity.type]))
        print(u'{:<16}: {}'.format('metadata', entity.metadata))
        print(u'{:<16}: {}'.format('salience', entity.salience))
        print(u'{:<16}: {}'.format('wikipedia_url',
              entity.metadata.get('wikipedia_url', '-')))
        
    return entities
text=u'<NAME>: One of the most useful things we learnt at ISS was to be analytical and to be able to adapt, \
       especially in an industry that is changing so quickly. What ISS gave us was an invaluable reference point.'
cloud_response = didi_entities_text(text)
print('[ cloud_response ]\n{}'.format(cloud_response))<jupyter_output>[ cloud_response ]
[name: "<NAME>"
type: PERSON
salience: 0.6122507452964783
mentions {
  text {
    content: "<NAME>"
    begin_offset: -1
  }
  type: PROPER
}
mentions {
  text {
    content: "One"
    begin_offset: -1
  }
  type: COMMON
}
, name: "industry"
type: OTHER
salience: 0.1542702317237854
mentions {
  text {
    content: "industry"
    begin_offset: -1
  }
  type: COMMON
}
, name: "things"
type: OTHER
salience: 0.12234412133693695
mentions {
  text {
    content: "things"
    begin_offset: -1
  }
  type: COMMON
}
, name: "ISS"
type: ORGANIZATION
metadata {
  key: "mid"
  value: "/m/03wky"
}
metadata {
  key: "wikipedia_url"
  value: "https://en.wikipedia.org/wiki/International_Space_Station"
}
salience: 0.08893194049596786
mentions {
  text {
    content: "ISS"
    begin_offset: -1
  }
  type: PROPER
}
mentions {
  text {
    content: "ISS"
    begin_offset: -1
  }
  type: PROPER
}
, name: "reference point"
type: OTHER
salience: 0.022202959284186363
mentions {
  tex[...]<jupyter_text>* 语句的语法分析 (Analyzing Syntax: syntactic analysis)<jupyter_code>def didi_syntax_text(text):
    """Detects syntax in the text."""
##################################################################
# (1) Instantiates a client - using GOOGLE_APPLICATION_CREDENTIALS
#     client_language = language.LanguageServiceClient()
# 
# (2) Instantiates a client - using 'service account json' file
    client_language = language.LanguageServiceClient.from_service_account_json(
        "/media/sf_vm_shared_folder/000-cloud-api-key/mtech-ai-7b7e049cf5f6.json")
##################################################################

    if isinstance(text, six.binary_type):
        text = text.decode('utf-8')

    # Instantiates a plain text document.
    document = types.Document(
        content=text,
        type=enums.Document.Type.PLAIN_TEXT)

    # Detects syntax in the document. You can also analyze HTML with:
    #   document.type == enums.Document.Type.HTML
    tokens = client_language.analyze_syntax(document).tokens

    # part-of-speech tags from enums.PartOfSpeech.Tag
    pos_tag = ('UNKNOWN', 'ADJ', 'ADP', 'ADV', 'CONJ', 'DET', 'NOUN', 'NUM',
               'PRON', 'PRT', 'PUNCT', 'VERB', 'X', 'AFFIX')

    for token in tokens:
        print(u'{}: {}'.format(pos_tag[token.part_of_speech.tag],
                               token.text.content))
        
    return tokens

text=u'<NAME>: One of the most useful things we learnt at ISS was to be analytical and to be able to adapt, \
       especially in an industry that is changing so quickly. What ISS gave us was an invaluable reference point.'
cloud_response = didi_syntax_text(text)
print('[ cloud_response ]\n{}'.format(cloud_response))<jupyter_output>[ cloud_response ]
[text {
  content: "Alvin"
  begin_offset: -1
}
part_of_speech {
  tag: NOUN
  number: SINGULAR
  proper: PROPER
}
dependency_edge {
  head_token_index: 1
  label: NN
}
lemma: "Alvin"
, text {
  content: "Khoo"
  begin_offset: -1
}
part_of_speech {
  tag: NOUN
  number: SINGULAR
  proper: PROPER
}
dependency_edge {
  head_token_index: 13
  label: NSUBJ
}
lemma: "Khoo"
, text {
  content: ":"
  begin_offset: -1
}
part_of_speech {
  tag: PUNCT
}
dependency_edge {
  head_token_index: 1
  label: P
}
lemma: ":"
, text {
  content: "One"
  begin_offset: -1
}
part_of_speech {
  tag: NUM
}
dependency_edge {
  head_token_index: 1
  label: APPOS
}
lemma: "One"
, text {
  content: "of"
  begin_offset: -1
}
part_of_speech {
  tag: ADP
}
dependency_edge {
  head_token_index: 3
  label: PREP
}
lemma: "of"
, text {
  content: "the"
  begin_offset: -1
}
part_of_speech {
  tag: DET
}
dependency_edge {
  head_token_index: 8
  label: DET
}
lemma: "the"
, text {
  content: "most"
  begi[...]<jupyter_text>* 消息文字的主题分类 (Classifying Content: analyze content to determine the content category)<jupyter_code>def didi_classify_text(text):
    """Classifies content categories of the provided text."""
##################################################################
# (1) Instantiates a client - using GOOGLE_APPLICATION_CREDENTIALS
#     client_language = language.LanguageServiceClient()
# 
# (2) Instantiates a client - using 'service account json' file
    client_language = language.LanguageServiceClient.from_service_account_json(
        "/media/sf_vm_shared_folder/000-cloud-api-key/mtech-ai-7b7e049cf5f6.json")
##################################################################

    if isinstance(text, six.binary_type):
        text = text.decode('utf-8')

    document = types.Document(
        content=text.encode('utf-8'),
        type=enums.Document.Type.PLAIN_TEXT)

    categories = client_language.classify_text(document).categories

    for category in categories:
        print(u'=' * 20)
        print(u'{:<16}: {}'.format('name', category.name))
        print(u'{:<16}: {}'.format('confidence', category.confidence))
        
    return categories
text=u'<NAME>: One of the most useful things we learnt at ISS was to be analytical and to be able to adapt, \
       especially in an industry that is changing so quickly. What ISS gave us was an invaluable reference point.'
cloud_response = didi_classify_text(text)
print('[ cloud_response ]\n{}'.format(cloud_response))
text=u'<NAME> (Sam) lectures Master of Technology programme in the areas of data science, machine intelligence, \
       soft computing, and applied deep learning. Prior to joining ISS, he was in New Zealand running Kudos Data \
       start-up, which focussed on training programs to democratize artificial intelligence and machine learning. \
       Before that, he was a data scientist in Yokogawa Engineering, leading industrial innovation initiatives. \
       Sam had also spent many years in financial sector wearing versatile hats: project manager, consultant, \
       and business analyst in Barclays bank, system manager and senior software engineer at Citibank, leading \
       to experience in actualizing information technology values in complex business environment. \
       He devotes himself into pedagogy, and is very passionate in inspiring next generation of \
       artificial intelligence lovers and leaders.'
cloud_response = didi_classify_text(text)
print('[ cloud_response ]\n{}'.format(cloud_response))<jupyter_output>[ cloud_response ]
[name: "/Jobs & Education/Education"
confidence: 0.75
, name: "/Science/Computer Science"
confidence: 0.6800000071525574
, name: "/Science/Engineering & Technology"
confidence: 0.550000011920929
]<jupyter_text>Wrap cloud APIs into Functions() for conversational virtual assistant (VA):Reuse above defined Functions().<jupyter_code>def didi_language_processing(text):
    nlp_reply  = u'[ NLP 自然语言处理结果 ]\n\n'
    nlp_reply += u'[ didi_sentiment_text ]\n\n' + str(didi_sentiment_text(text)) + u'\n\n'
    nlp_reply += u'[ didi_entity_sentiment_text ]\n\n' + str(didi_entity_sentiment_text(text)) + u'\n\n'
    nlp_reply += u'[ didi_entities_text ]\n\n' + str(didi_entities_text(text)) + u'\n\n'
    nlp_reply += u'[ didi_syntax_text ]\n\n' + str(didi_syntax_text(text)) + u'\n\n'
    nlp_reply += u'[ didi_classify_text ]\n\n' + str(didi_classify_text(text)) + u'\n\n'
    
    return nlp_reply
demo_text = 'Sam devotes himself into pedagogy, and is very passionate in inspiring next generation of \
artificial intelligence lovers and leaders.'

demo_nlp_reply = didi_language_processing(demo_text)
print(demo_nlp_reply)
# Comment some API which does not support Chinese language:
def didi_language_processing_chinese(text):
    nlp_reply  = u'[ NLP 自然语言处理结果 ]\n\n'
    nlp_reply += u'[ didi_sentiment_text ]\n\n' + str(didi_sentiment_text(text)) + u'\n\n'
#     nlp_reply += u'[ didi_entity_sentiment_text ]\n\n' + str(didi_entity_sentiment_text(text)) + u'\n\n'
    nlp_reply += u'[ didi_entities_text ]\n\n' + str(didi_entities_text(text)) + u'\n\n'
    nlp_reply += u'[ didi_syntax_text ]\n\n' + str(didi_syntax_text(text)) + u'\n\n'
#     nlp_reply += u'[ didi_classify_text ]\n\n' + str(didi_classify_text(text)) + u'\n\n'
    
    return nlp_reply
demo_text = '山姆致力于教育学，并且非常热衷于激励下一代人工智能爱好者和领导者。'

demo_nlp_reply = didi_language_processing_chinese(demo_text)
print(demo_nlp_reply)<jupyter_output>[ NLP 自然语言处理结果 ]

[ didi_sentiment_text ]

magnitude: 0.30000001192092896
score: 0.30000001192092896


[ didi_entities_text ]

[name: "\345\261\261\345\247\206"
type: PERSON
salience: 0.43622928857803345
mentions {
  text {
    content: "\345\261\261\345\247\206"
    begin_offset: -1
  }
  type: PROPER
}
, name: "\346\225\231\350\202\262\345\255\246"
type: OTHER
salience: 0.29734984040260315
mentions {
  text {
    content: "\346\225\231\350\202\262\345\255\246"
    begin_offset: -1
  }
  type: COMMON
}
, name: "\351\242\206\345\257\274\350\200\205"
type: PERSON
salience: 0.13476282358169556
mentions {
  text {
    content: "\351\242\206\345\257\274\350\200\205"
    begin_offset: -1
  }
  type: COMMON
}
, name: "\346\231\272\350\203\275\347\210\261\345\245\275\350\200\205"
type: PERSON
salience: 0.13165807723999023
mentions {
  text {
    content: "\346\231\272\350\203\275\347\210\261\345\245\275\350\200\205"
    begin_offset: -1
  }
  type: COMMON
}
]

[ didi_syntax_text ]

[text {
  [...]<jupyter_text>Start interactive conversational virtual assistant (VA):  Import ItChat, etc. 导入需要用到的一些功能程序库：<jupyter_code># import io, os, subprocess, sys, re, codecs, time, datetime, requests, itchat
import itchat
from itchat.content import *<jupyter_output><jupyter_text>Log in using QR code image / 用微信App扫QR码图片来自动登录<jupyter_code># itchat.auto_login(hotReload=True) # hotReload=True: 退出程序后暂存登陆状态。即使程序关闭，一定时间内重新开启也可以不用重新扫码。
itchat.auto_login(enableCmdQR=-2) # enableCmdQR=-2: 命令行显示QR图片
# Obtain my own Nick Name
MySelf = itchat.search_friends()
NickName4RegEx = '@' + MySelf['NickName'] + '\s*'
# 单聊模式，自动进行自然语言分析，以文本形式返回处理结果：
@itchat.msg_register([TEXT, MAP, CARD, NOTE, SHARING])
def text_reply(msg):
        text = msg['Content']
        # call NLP API:
        nlp_responses = didi_language_processing(text)
#         nlp_responses = didi_language_processing_chinese(text)
        # Format NLP results: 
        nlp_reply = nlp_responses # Exercise / Workshop Enhancement: to parse and format results nicely.
        print(nlp_reply)
        return nlp_reply
# 群聊模式，如果收到 @ 自己的文字信息，会自动进行自然语言分析，以文本形式返回处理结果：
@itchat.msg_register(TEXT, isGroupChat=True)
def text_reply(msg):
    if msg['isAt']:
        text = re.sub(NickName4RegEx, '', msg['Content'])
        # call NLP API:
        nlp_responses = didi_language_processing(text)
#         nlp_responses = didi_language_processing_chinese(text)
        # Format NLP results:
        nlp_reply = nlp_responses # Exercise / Workshop Enhancement: to parse and format results nicely.
        print(nlp_reply)
        return nlp_reply
itchat.run()
# interupt kernel, then logout
itchat.logout() # 安全退出<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14562524970035956, 'fraction_numerical': 0.04239912105473432, 'mean_word_length': 4.094123632663444, 'pattern_counts': {'":': 2, '<': 56, '<?xml version=': 0, '>': 51, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '550219', 'license': 'MIT', 'n_tokens_mistral': 7317, 'n_tokens_neox': 6716, 'n_words': 1601, 'path': 'wechat_tool_py3_local/lesson_4_py3_local.ipynb', 'repo_name': 'telescopeuser/workshop_blog_py3_local', 'size': '20062'}