<jupyter_start><jupyter_text>HOCOR2020 Session 1.3 資料前處理  環境設定<jupyter_code>pip install gdown zhon 
!gdown --id 11wLV1yLtj5ot5zRQmAh8gk6Ndr06k0_K -O data_1.3.zip && unzip data_1.3.zip
import pickle
with open("data_1.3/wiki_pages.pkl", "rb") as fin:
    data = pickle.load(fin)
list(data.items())[:3]<jupyter_output><empty_output><jupyter_text>資料整理 * 把資料整理成一個List of Dict，每個Dict是一篇「文章」。文章有三個「屬性」：查詢詞(`query`)、標題(`title`)、原始材料(`rawtext`)<jupyter_code>corpus = []
for key, data_item in data.items():
    article = {}
    article["query"] = key
    article["title"] = data_item[0]
    article["rawtext"] = data_item[1]
    corpus.append(article)
corpus[:3]<jupyter_output><empty_output><jupyter_text>簡單的描述性資料<jupyter_code>n_char_raw = sum(len(x["rawtext"]) for x in corpus)
print("文章個數: ", len(corpus))
print("文章總長度: ", n_char_raw)
print("平均文章長度: ", round(n_char/len(corpus), 2))
zhon.hanzi.characters
import re
import zhon.hanzi
zh_pat = f"[{zhon.hanzi.characters}|{zhon.hanzi.punctuation}]"
for article in corpus:
    zhstream = "".join(re.findall(zh_pat, article["rawtext"]))
    article["preproc"] = re.findall(zhon.hanzi.sentence, zhstream)
corpus[:2]
n_char_preproc = sum([sum(len(y) for y in x["preproc"]) for x in corpus])
print("文章原始總長度: ", n_char_raw)
print("文章前處理後總長度: ", n_char_preproc)
print("前處理保留比例: ", round(n_char_preproc/n_char_raw, 2))<jupyter_output>文章原始總長度:  40317
文章前處理後總長度:  27415
前處理保留比例:  0.68<jupyter_text>檔案儲存  純文字<jupyter_code>with open("corpus-wiki.txt", "w", encoding="UTF-8") as fout:
    for article in corpus:
        fout.write("\n".join(article["preproc"]))
        fout.write("\n")<jupyter_output><empty_output><jupyter_text>逗號分隔檔<jupyter_code>import csv
with open("corpus-wiki.csv", "w", encoding="UTF-8") as fout:
    writer = csv.writer(fout)
    writer.writerow(["query", "title", "preproc"])
    for article in corpus:
        writer.writerow([article["query"], article["title"], "\n".join(article["preproc"])])<jupyter_output><empty_output><jupyter_text>JSON格式<jupyter_code>import json
with open("corpus-wiki.json", "w", encoding="UTF-8") as fout:
    json.dump(corpus, fout, ensure_ascii=False, indent=2)

print(json.dumps(corpus[0], ensure_ascii=False, indent=2))
print(json.dumps(corpus[0], ensure_ascii=False, indent=None))
print(json.dumps(corpus[0], ensure_ascii=True, indent=None))<jupyter_output>{"query": "\u8a9e\u97f3\u5b78", "title": "\u8a9e\u97f3\u5b78", "rawtext": "\u8a9e\u97f3\u5b78\uff08Phonetics\u3001\u767c\u97f3\uff1a/f\u0259\u02c8n\u025bt\u026aks/\uff09\u662f\u7814\u7a76\u8a00\u8a9e\u8072\u97f3\uff08\u5373\u8a9e\u97f3\uff09\u7684\u8a9e\u8a00\u5b78\u5206\u652f\u5b78\u79d1\u3002\u72f9\u7fa9\u7684\u8a9e\u97f3\u5b78\u5c0d\u61c9\u82f1\u8a9e\u4e2dphonetics(\u767c\u97f3)\u4e00\u8a5e\uff0c\u95dc\u5207\u7684\u91cd\u9ede\u5728\u8a9e\u97f3\u7684\u7269\u7406\u3001\u751f\u7269\u3001\u5fc3\u7406\u7b49\u5177\u8c61\u672c\u8cea\uff0c\u8207\u8a9e\u97f3\u7684\u5be6\u969b\u7528\u9014\u3001\u8868\u9054\u610f\u7fa9\u7121\u95dc\u3002\u8207\u4e4b\u76f8\u5c0d\u7684\u662f\u97f3\u97fb\u5b78\uff08\u6216\u7a31\u97f3\u7cfb\u5b78\uff09\uff0c\u7814\u7a76\u97f3\u4f4d\u6216\u8a9e\u97f3\u5340\u5225\u7279\u5fb5\u5728\u67d0\u7a2e\u8a9e\u8a00\u4e2d\u904b\u4f5c\u7684\u62bd\u8c61\u898f\u5247\u548c\u8a9e\u97f3\u7684\u7cfb\u7d71\u3002\u5ee3\u7fa9\u7684\u8a9e\u97f3\u5b78\u662f\u6307\u9019\u5169\u5927\u65b9\u97[...]
{'fraction_non_alphanumeric': 0.1759784881983866, 'fraction_numerical': 0.15147893636092022, 'mean_word_length': 6.138592750533049, 'pattern_counts': {'":': 3, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '567384', 'license': 'MIT', 'n_tokens_mistral': 2004, 'n_tokens_neox': 1829, 'n_words': 150, 'path': 'hocor2020/notebook/session-1.3.ipynb', 'repo_name': 'lopentu/Hands-on_Corpus_Linguistics', 'size': '3347'}