<jupyter_start><jupyter_text>作業目標: 使用python正規表達式對資料進行清洗處理這份作業我們會使用詐欺郵件的文本資料來作為清洗與處理的操作。[資料集](https://www.kaggle.com/rtatman/fraudulent-email-corpus/data)  讀入資料文本因原始文本較大，先使用部份擷取的**sample_emails.txt**來進行練習<jupyter_code>#讀取文本資料
with open("./sample_emails.txt", "r") as f:
    sample_corpus = f.read()
sample_corpus[:1000]<jupyter_output><empty_output><jupyter_text>讀取寄件者資訊觀察文本資料可以發現, 寄件者資訊都符合以下格式`From:  `<jupyter_code>#<your code>#
import re
match = re.findall(pattern="From:.*.com\>", string=sample_corpus)
match<jupyter_output><empty_output><jupyter_text>只讀取寄件者姓名<jupyter_code>for line in match:
    search_obj = re.search(pattern="\".+\"", string=line)
    if search_obj: print(search_obj.group())<jupyter_output>"MR. <NAME>."
"Mr. <NAME>"
"<NAME>"<jupyter_text>只讀取寄件者電子信箱<jupyter_code>emails = []
for line in match:
    search_obj = re.search(pattern="\w+@.+\.com", string=line)
    if search_obj: 
        email = search_obj.group()
        emails.append(email)
        print(email)<jupyter_output><EMAIL>
<EMAIL>
<EMAIL><jupyter_text>只讀取電子信箱中的寄件機構資訊ex: <EMAIL> --> 取maktoob<jupyter_code>for email in emails:
    search_obj = re.search(pattern="(?<=@)\w+(?=\.com)", string=email)
    if search_obj:
        print(search_obj.group())<jupyter_output>maktoob
spinfinder
epatra<jupyter_text>結合上面的配對方式, 將寄件者的帳號與機構訊返回ex: <EMAIL> --> [james_ngola2002, maktoob]<jupyter_code>for email in emails:
    findall_obj = re.findall(pattern=".+(?=\.com)", string=email)
    if len(findall_obj):
        ta, tb = re.split("@", findall_obj[0])
        print(f"{ta}, {tb}")<jupyter_output>james_ngola2002, maktoob
bensul2004nng, spinfinder
obong_715, epatra<jupyter_text>使用正規表達式對email資料進行處理這裡我們會使用到python其他的套件協助處理(ex: pandas, email, etc)，這裡我們只需要專注在正規表達式上即可，其他的套件是方便我們整理與處理資料。  讀取與切分Email讀入的email為一個長字串，利用正規表達式切割讀入的資料成一封一封的email，並將結果以list表示。輸出: [email_1, email_2, email_3, ....]<jupyter_code>###讀取文本資料:fradulent_emails.txt###
with open('all_emails.txt', 'r', encoding="utf-8", errors='ignore') as f:
    corpus = f.read()
    
###切割讀入的資料成一封一封的email###
emails = re.split(pattern="From r", string=corpus)

###我們可以使用list來儲存每一封email###
###注意！這裡請仔細觀察sample資料，看資料是如何切分不同email###
#<your code>#
emails = emails[1:]
len(emails) #查看有多少封email<jupyter_output><empty_output><jupyter_text>從文本中擷取所有寄件者與收件者的姓名和地址<jupyter_code>import re
import pandas as pd
import email

emails_list = [] #創建空list來儲存所有email資訊

for mail in emails[:20]: #只取前20筆資料 (處理速度比較快)
    emails_dict = dict() #創建空字典儲存資訊
    ###取的寄件者姓名與地址###
    
    #Step1: 取的寄件者資訊 (hint: From:)
    search_obj_sender = re.search("(?<=From\:).*\<.*\>", mail)
    
    #Step2: 取的姓名與地址 (hint: 要注意有時會有沒取到配對的情況)
    if search_obj_sender:
        search_obj_name = re.search("(?<=\").+(?=\")", search_obj_sender.group())
        search_obj_mail = re.search("(?<=\<).+(?=\>)", search_obj_sender.group())
        
        #Step3: 將取得的姓名與地址存入字典中
        if search_obj_name and search_obj_mail:
            emails_dict['sender_name'] = search_obj_name.group()
            emails_dict['sender_mail'] = search_obj_mail.group()
        else:
            emails_dict['sender_name'] = None
            emails_dict['sender_mail'] = None

    ###取的收件者姓名與地址###
    
    #Step1: 取的寄件者資訊 (hint: To:)
    search_obj_recipient = re.search("(?<=\nTo:).+", mail)
    
    #Step2: 取的姓名與地址 (hint: 要注意有時會有沒取到配對的情況)
    #Step3: 將取得的姓名與地址存入字典中
    if search_obj_recipient:
        emails_dict['recipient_name'] = search_obj_recipient.group().strip().split("@")[0]
        emails_dict['recipient_mail'] = search_obj_recipient.group().strip()
    else:
        emails_dict['recipient_name'] = None
        emails_dict['recipient_mail'] = None
        
    ###取得信件日期###
    #Step1: 取得日期資訊 (hint: To:)
    search_obj_date = re.search("(?<=Date).*", mail)

    #Step2: 取得詳細日期(只需取得DD MMM YYYY)
    #Step3: 將取得的日期資訊存入字典中
    if search_obj_date:
        data_info = search_obj_date.group()
        emails_dict['date'] = re.search("\d+\s\w+\s\d+", data_info).group()
    else:
        emails_dict['date'] = None
        
    ###取得信件主旨###
    #Step1: 取得主旨資訊 (hint: Subject:)
    search_obj_subject = re.search("(?<=Subject:\s).*", mail)
    
    #Step2: 移除不必要文字 (hint: Subject: )
    #Step3: 將取得的主旨存入字典中
    if search_obj_subject:
        emails_dict['subject']= search_obj_subject.group()
    else:
        emails_dict['subject']= None
    
    ###取得信件內文###
    #這裡我們使用email package來取出email內文 (可以不需深究，本章節重點在正規表達式)
    try:
        full_email = email.message_from_string(mail)
        body = full_email.get_payload()
        emails_dict["email_body"] = body
    except:
        emails_dict["email_body"] = None
    
    ###將字典加入list###
    emails_list.append(emails_dict)
#將處理結果轉化為dataframe
emails_df = pd.DataFrame(emails_list)
emails_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1574405388339297, 'fraction_numerical': 0.008840244159124395, 'mean_word_length': 4.363431151241535, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 44, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '478804', 'license': 'MIT', 'n_tokens_mistral': 2314, 'n_tokens_neox': 2428, 'n_words': 257, 'path': 'practices/Day-004/day_004_practice.ipynb', 'repo_name': 'magikerwin1993/1st-NLP-Marathon', 'size': '4867'}