<jupyter_start><jupyter_text>Week 5. In-class Practice  TF-IDF練習:使用華航載運iphone 8 新機的相關新聞<jupyter_code>from collections import Counter
import jieba
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import pandas as pd
import sys
sys.path.insert(0, '../modules/')

from fintextmin import *  # 使用 modules 中的 fintextmin 模組<jupyter_output><empty_output><jupyter_text>從指定路徑中讀入新聞資料<jupyter_code>text_list = read_text_folder("./dataset/")
print("document number:", len(text_list))<jupyter_output>document number: 9<jupyter_text>製作詞頻矩陣下面以 Python 實現製作詞頻矩陣的步驟，並附上對應的`fintextmin`函式<jupyter_code>## [原生 python 寫法]
doc_names = range(len(text_list))
dwc = {key: [word for word in jieba.cut(text, cut_all=False)] for (key, text) in zip(doc_names, text_list)}


## [使用 fintextmin 模組]
# stopwords = [] 
# dwc = count_words_in_documents(doc_list = text_list,
#                               doc_names = range(len(text_list)),
#                                stopwords = stopwords)
## [原生 python 寫法] 
doc_word_frequency = {key: {i:Counter(doc)[i] for i in Counter(doc)} for key, doc in dwc.items()}
list(doc_word_frequency[0].items())[0:10] # 顯示第一篇文章的部份詞彙與對應詞頻

## [使用 fintextmin 寫法]
# doc_word_frequency = create_word_frequency_matrix(dwc)
## [原生 python 寫法] 
dwf_list = [pd.DataFrame(list(doc.values()), index=doc.keys()) for doc in doc_word_frequency.values()]
wfm = pd.concat(dwf_list, axis= 1)
wfm = np.transpose(wfm).fillna(0)
wfm.index = doc_word_frequency.keys()

puncs = set_puncs() # 因為一開始沒有清除標點符號,這邊回過頭來將它們自矩陣移除
wfm = wfm[list(set(wfm) - set(puncs))]
wfm # 詞頻矩陣完成

## [使用 fintextmin 寫法]
# wfm = create_word_frequency_matrix(dwc)<jupyter_output><empty_output><jupyter_text>製作TF-IDF矩陣可以將完成後的tf-idf矩陣與上面的詞頻矩陣比較一下係數<jupyter_code>from sklearn import feature_extraction
from sklearn.feature_extraction.text import TfidfTransformer

## [原生 python 寫法] 
transformer = TfidfTransformer()
tfidf_gen = transformer.fit_transform(wfm) # 產生一個 tfidf 矩陣的 generator
df_tfidf = pd.DataFrame(tfidf_gen.toarray(), columns=wfm.columns)
df_tfidf

## [使用 fintextmin 寫法]
# df_tfidf = evaluate_tfidf(wfm)<jupyter_output><empty_output><jupyter_text>視覺化TFIDF和原本詞頻矩陣的差異 這邊我們取第一篇文章來比較看看調整前後的詞彙係數<jupyter_code>first_text = wfm.to_dict(orient='records')[0] # 將詞頻矩陣轉換成dictionary型別, 把第一篇文章抽出來
list(first_text.items())[0:10] # 檢視一下轉換結果<jupyter_output><empty_output><jupyter_text>文字雲<jupyter_code>linux_font = r'/usr/share/fonts/truetype/wqy/wqy-microhei.ttc'

plt.figure(figsize=(12, 6))
ax1 = plt.subplot(121)
plot_wordcloud(first_text, ax=ax1, font_path=linux_font, height=300, width=300)

ax2 = plt.subplot(122)
plot_tfidf_wordcloud(df_tfidf, text_index=0, ax=ax2, font_path=linux_font, height=300, width=300)<jupyter_output><empty_output><jupyter_text>長條圖可以發現，經過TF-IDF處理之後，比較重要的資訊如「股價」、「747」(註：此指波音747)等詞彙被突顯出來。每篇文章都會出現的「華航」和所有格用字「的」排序則向下調整。<jupyter_code>plt.figure(figsize=(18, 5))
ax1 = plt.subplot(121)
plot_tf_bar(first_text, ax=ax1, font_path=linux_font, title="詞頻")

ax2 = plt.subplot(122)
plot_tfidf_bar(df_tfidf, text_index=0, ax=ax2, font_path=linux_font, title="tf-idf")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13143412107478147, 'fraction_numerical': 0.019423761735189383, 'mean_word_length': 7.24, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '648052', 'license': 'MIT', 'n_tokens_mistral': 1611, 'n_tokens_neox': 1608, 'n_words': 194, 'path': 'week_05/week_5_inclass_practice.ipynb', 'repo_name': 'kevinkevin556/STASD', 'size': '3089'}