<jupyter_start><jupyter_text>---  Review of Data Collection  ---  Natural Language Processing  We will demonstrate how to go through these five steps for English and Chinese texts respectively.  1. Data Cleaning- Main task: convert the case, remove punctuations and special characters like hashtags, hyperlinks- Use Regular Expression for Pattern Matching- Convert the case: `.lower()`  Regular Expression Cheat Sheet- `.` matches any single character- `[...]` group matching, matches any one of the characters inside the square brackets- `[^x]` matches one character that is not x- `|` an “or” operator, matches patterns on either side of the |.- `*` matches at least 0 times.- `+` matches at least 1 times.- `?` matches at most 1 times.- `{n}` matches n times- `(...)` grouping in regular expressions- `\\N` – backreference to group N- `^` matches the start of the string.- `$` matches the end of the string. 1a. English<jupyter_code>#install regular expression package for pattern matching
! pip3 install regex
import regex as re
#Use sub() function to match pattern and substitute the matched words with new pattern
a='I only have 100 dollars in my pocket. What I can buy?'
re.sub('I','You',a) #substitute a word with another word
#Substitute a word with nothing, meaning removing the word
re.sub('I','',a)
#Match a group of words by using []. "a-z" means all capital letters.
re.sub('[A-Z]','*',a)
#Remove all letters in lower case
re.sub('[a-z]','',a)
#Hide the numbers
re.sub('[0-9]','*',a)
#Remove all alphanumeric characters
re.sub('[0-9a-zA-Z]','',a)
#Remove all charaters that are not alphanumeric
re.sub('[^0-9a-zA-Z]',' ',a)
#Shortcut to remove all punctuation
re.sub('\p{P}+',' ',a) #\p stands for POSIX characters. {P} stands for punctuation.
#Remove hashtags
a='''@JerryNadler admits on #CNN they have no proof of Obstruction by @realDonaldTrump it's just his "personal opinion" Meet the new #WitchHunt Same as the old #WitchHunt cc @DonaldJTrumpJr'''
re.sub('#[^ ]+','',a)
#Extract hashtags by findall() function
re.findall('#[^ ]+',a)
#Extract all mentions
re.findall('@[^ ]+',a)
#Remove hyperlinks
a='Tesla’s abrupt shift to online-only car sales, after racing to open stores, battered its share price and raised questions about its future. https://goo.gl/rwGHTP'
re.sub('https://[^ ]+|http://[^ ]+','',a)
#tranform all letters to lower case
a.lower()<jupyter_output><empty_output><jupyter_text>Practice Create a data_cleaning() function to convert letter case, remove punctuations, numbers, mentions, hashtags and hyperlinks<jupyter_code>def data_cleaning(text):
    text=text.lower()
    text=re.sub('[0-9]+','',text)
    text=re.sub('@[^ ]+','',text)
    text=re.sub('#[^ ]+','',text)
    text=re.sub('http://[^ ]+|https://[^ ]+','',text)
    text=re.sub('\p{P}+',' ',text)
    return(text)
#test your function with a post from @realDonaldTrump
a='@seanhannity “We the people will now be subjected to the biggest display of modern day McCarthyism....which is the widest fishing net expedition....every aspect of the presidents life....all in order to get power back so they can institute Socialism.” https://t.co/izb2tTrINB'
data_cleaning(a)
re.sub('\p{P}+',' ',re.sub('http://[^ ]+|https://[^ ]+|@[^ ]+|#[^ ]+','',a))<jupyter_output><empty_output><jupyter_text>--- Break---  Tokenization- Definition: tokenization is a process of splitting sentences/paragraphs/documents into a set of words.- Differences in Languages:    - English: **words** are naturally separated with spaces    - Korean: **phrases** are naturally separated with spaces        - konlpy (http://konlpy.org/)    - Chinese/Japanese: **no spaces** in text        - Chinese: jieba (https://github.com/fxsjy/jieba)        - Japanese: jNlp (https://github.com/kevincobain2000/jProcessing)  Tokenize English Text: Hunt for Spaces<jupyter_code>#Split the following sentence into words
sentence='Mr. Zuckerberg, who runs Facebook, Instagram, WhatsApp and Messenger, on Wednesday expressed his intentions to change the essential nature of social media. Instead of encouraging public posts, he said he would focus on private and encrypted communications, in which users message mostly smaller groups of people they know. Unlike publicly shared posts that are kept as users’ permanent records, the communications could also be deleted after a certain period of time.'
sentence=data_cleaning(sentence)
words=sentence.split(' ')
import pandas as pd<jupyter_output><empty_output><jupyter_text>**Extra Knowledge** We can use funtion gensim.parsing.preprocessing.stem_text(text) to stem words in the sentence.   Tokenize Chinese Text We will use a package package "jieba" to tokenize Chinese text.**Why jieba?**- It adopts a hybrid method combining both statistical/probabilistic inference and pattern matching based on dictionary.     - capable to recognize words existing in the pre-defined dictionary    - capable to find new words.- Two dictionaries:    - System dictionary        - Simplied Chinese        - Simplied+Traditional Chinese    - User dictionary<jupyter_code>! pip3 install jieba
import jieba
list(jieba.cut('你好，这是一个简单的句子。'))
#it can segment tradional Chinese text by using statistical inference method.
list(jieba.cut('你好，這是一個簡單的句子。'))
#however, statistical inference is not perfect.
list(jieba.cut('談判擱置，工會號召靜坐。'))
list(jieba.cut('谈判搁置，工会号召静坐。'))<jupyter_output><empty_output><jupyter_text>Configurate Dictionaries To better segment traditional Chinese text, we need to upgrade system dictionary to include traditional Chinese words.Download the system dictionary from this link:https://github.com/fxsjy/jieba/raw/master/extra_dict/dict.txt.big<jupyter_code>#load another dictionary to support traditional Chinese
jieba.set_dictionary('C:\\Users\\yuner\\AppData\\Local\\Programs\\Python\\Python36\\Lib\\site-packages\\jieba\\dict.txt.big')
#try tokenizing this sentence again
list(jieba.cut('談判擱置，工會號召靜坐。'))
#Some names and special terminologies cannot be properly identified.
print(list(jieba.cut('中央上周二向特首林鄭月娥發公函'))) #very long name
print(list(jieba.cut('台灣蔡英文總統日前表示希望與日本舉行安保對話'))) #names including frequently used words
print(list(jieba.cut('高雄市長韓國瑜本月稍後訪問港澳深圳廈門四市'))) #names including frequently used words
print(list(jieba.cut('汶萊的全稱為汶萊達魯薩蘭國。'))) #special terminologies
#Build your user dictionary (time-consuming)
file=open('user_dict.txt','w',encoding='utf-8')
file.write('林鄭月娥\n')
file.write('蔡英文\n')
file.write('韓國瑜\n')
file.write('汶萊達魯薩蘭國\n')
file.close()
#Use your user dictionary
jieba.load_userdict('user_dict.txt')
#After loading user dictionary:
print(list(jieba.cut('中央上周二向特首林鄭月娥發公函'))) #very long name
print(list(jieba.cut('台灣蔡英文總統日前表示希望與日本舉行安保對話'))) #names including frequently used words
print(list(jieba.cut('高雄市長韓國瑜本月稍後訪問港澳深圳廈門四市'))) #names including frequently used words
print(list(jieba.cut('汶萊的全稱為汶萊達魯薩蘭國。'))) #terminologies<jupyter_output>['中央', '上周二', '向', '特首', '林鄭月娥', '發', '公函']
['台灣', '蔡英文', '總統', '日前', '表示', '希望', '與', '日本', '舉行', '安保', '對話']
['高雄', '市長', '韓國瑜', '本月', '稍後', '訪問', '港澳', '深圳', '廈門', '四市']
['汶萊', '的', '全稱', '為', '汶萊達魯薩蘭國', '。']<jupyter_text>Remove stop wordsStop words are useless for understanding text.In English: at, in, on, for, of, a, an, the...In Chinese: 的，地，得，了. However, the combination of 不得了 (holy great) is not a stop word which is used to convey extreme compliment over something.√ Absolute Match. × Pattern Matching<jupyter_code>'a' in ['a','b','c']
'a' in ['aa','b','c']
'a' not in ['aa','b','c']<jupyter_output><empty_output><jupyter_text>Chinese stop words file: https://juniorworld.github.io/python-workshop-2018/doc/stop_words_chi.txtEnglish stop words file: https://juniorworld.github.io/python-workshop-2018/doc/stop_words_eng.txt<jupyter_code>file_chi=open('C:\\Users\\yuner\\AppData\\Local\\Programs\\Python\\Python36\\Lib\\site-packages\\jieba\\stop_words_chi.txt','r',encoding='utf-8')
stop_words_chi=[]
for line in file_chi.readlines():
    line=line.strip() #remove line break
    stop_words_chi.append(line) #update the list of stop words line by line
file_chi.close()
len(stop_words_chi)
file_eng=open('C:\\Users\\yuner\\AppData\\Local\\Programs\\Python\\Python36\\Lib\\site-packages\\jieba\\stop_words_eng.txt','r')
stop_words_eng=[]
for line in file_eng.readlines():
    line=line.strip() #remove line break
    stop_words_eng.append(line) #update the list of stop words line by line
file_eng.close()
len(stop_words_eng)<jupyter_output><empty_output><jupyter_text>Absolute Match of Stop words<jupyter_code>sentence='Facebook将向加密通信转型，打造以隐私为中心的平台。'
words=list(jieba.cut(sentence))
words_new=[]
for word in words:
    if word not in stop_words_chi:
        words_new.append(word)
words_new
#for loop in the list
a=[1,2,3,4,5]
b=[i+1 for i in a]           #increase element by one
#for loop and if statement in the list
a=[1,2,3,4,5]
b=[i for i in a if i<4]
b
words_new=[word for word in list(jieba.cut(sentence)) if word not in stop_words_chi]
words_new
#Clean and tokenize this sentence and remove the stop words
sentence='Mr. Zuckerberg, who runs Facebook, Instagram, WhatsApp and Messenger, on Wednesday expressed his intentions to change the essential nature of social media. Instead of encouraging public posts, he said he would focus on private and encrypted communications, in which users message mostly smaller groups of people they know. Unlike publicly shared posts that are kept as users’ permanent records, the communications could also be deleted after a certain period of time.'
words_new=[word for word in data_cleaning(sentence).split(' ') if word not in stop_words_eng]

words_new<jupyter_output><empty_output><jupyter_text>Practice Find the 10 fade-in and fade-out words in speeches.The magnitude of difference is measured by the change in their relative frequencies:Relative Freq (RF) = word frequency / max word frequencyDifference = RF2019 - RF2009 Options:- Chinese: Annual government work reports, 2019 vs 2009- English: State of the Union address, 2019 vs 2009*Hint:**1. You can use `pd.concat([df1,df2],axis=1)` to combine two data frames by columns**2. You can use `df.fillna(0)` to replace NAN value with 0.**3. You can use `df.sort_values(column_name)` to sort a certain column.*<jupyter_code>freq_words=pd.Series(words).value_counts()
freq_words/max(freq_words)
file_2019=open('doc/2019_Government_Work_Report.txt','r',encoding='utf-8')
file_2009=open('doc/2009_Government_Work_Report.txt','r',encoding='utf-8')
#CHI
words_new=[]
for line in file_2019.readlines():
    line=line.strip()
    line=data_cleaning(line)
    words=list(jieba.cut(line))
    for word in words:
        if word not in stop_words_chi:
            words_new.append(word)
#ENG
words_new=[]
for line in file_2019.readlines():
    line=line.strip()
    line=data_cleaning(line)
    words=line.split(' ')
    for word in words:
        if word not in stop_words_chi:
            words_new.append(word)
words_new_2009=[]
for line in file_2009.readlines():
    line=line.strip()
    line=data_cleaning(line)
    words=list(jieba.cut(line))
    for word in words:
        if word not in stop_words_chi:
            words_new_2009.append(word)
words_new_2009
freq_2019=pd.Series(words_new).value_counts()
freq_2009=pd.Series(words_new_2009).value_counts()
relative_freq_2019=freq_2019/max(freq_2019)
relative_freq_2009=freq_2009/max(freq_2009)
relative_freq=pd.concat([relative_freq_2019,relative_freq_2009],axis=1)
relative_freq=relative_freq.fillna(0)
relative_freq['diff']=relative_freq[0]-relative_freq[1]
relative_freq.columns=['2019','2009','diff']
relative_freq.sort_values('diff').tail(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1327639751552795, 'fraction_numerical': 0.015959282263630088, 'mean_word_length': 5.37328202308961, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 36, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '222564', 'license': 'MIT', 'n_tokens_mistral': 4166, 'n_tokens_neox': 4028, 'n_words': 1166, 'path': 'doc/Class7-updated.ipynb', 'repo_name': 'juniorworld/python-workshop-2018', 'size': '11592'}