<jupyter_start><jupyter_text>General GuidelineThis homework is basically an individual-oriented work. Each student has to doit all by himself (or herself). The final score will be evaluated from the systemperformance and individual demonstration. Homework OverviewImplement a full-text retrieval tool (i.e. search engine) for a set of textdocuments. In other words, your system will be able to perform document retrievalaccording to specified keyword(s) and display in an easily visualization way. Also, thetool is able to **calculate the document statistics (such as number of characters, numberof words, number of sentences(EOS), etc.) and to determine how many sentences indocuments using smart method (for example, rule-based approach)**. Computerlanguages are not restrictive. System Description1. This homework uses both the standard PubMed Biomedical Data which canbe obtained at http://www.ncbi.nlm.nih.gov/ under XML format and twitterdata at http://twitter.com/ under JSON format for information retrievalpurpose.2. Each individual student builds its own system components using basicmatching scheme from the IR course.3. Basic document preprocessing algorithms can be obtained from publicdomain (e.g. Porter’s algorithm, stop-words, stemming algorithm, etc.). In thefinal evaluation, each individual presents system description, running results,and system demo to verify performance.<jupyter_code>##  文件資料夾&文件名稱&對應的路徑
import os
doc_folder = 'hw1_data/' 
doc_name = os.listdir(doc_folder)
doc_link = [os.path.join(doc_folder, i) for i in doc_name]
##  先讀一篇xml出來看看
import xml.etree.ElementTree as ET
xml_doc = doc_link[3]
doc_tree = ET.parse(xml_doc)
doc_root = doc_tree.getroot()  ##  獲取根目錄的
doc_root.tag  ##  根目錄的 tag
element_list = [element for element in doc_tree.iter()]
content_string = " ".join([element.text for element in doc_tree.iter()])
content_token = word_tokenize(content_string)
keyword = "Rothe C"
content_token.count(keyword)
# key = "COVID"
# import re
# re.sub('[^{}]'.format(key), "", content_string)
# content_string
xml_doc
content_token
import nltk
from nltk.tokenize import word_tokenize
nltk.download('punkt', download_dir='./')
# nltk.download('punkt')
word_tokenize(content_string)
##  噴出所有的 element 物件，這樣對於找出所有 tag 中的內容有幫助
# doc_element_list = [element for element in doc_tree.iter()]
for index, element in enumerate(doc_tree.iter()):
    if(index==0):
        ##  這是父元素
        print(element.text)
        print(element.text=='\n')
    else:
        print(element)
        print(element.text=='\n    ')
        print(element.text=='\n        ')
#         x = element.text
#         if(index==1):
#             break

#     # break
# element.text=='\n    '
element.text == '\n'
##  目標：輸入一個關鍵字，輸入搜索文件格式，輸入文章的，輸出「這篇文章有幾個字元」、「這篇文章有幾個字詞」、「這篇文章有幾個句子」、<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10056858564321251, 'fraction_numerical': 0.0024875621890547263, 'mean_word_length': 4.663983903420523, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 5}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '441090', 'license': 'Apache-2.0', 'n_tokens_mistral': 933, 'n_tokens_neox': 940, 'n_words': 278, 'path': 'HW/1/skip/hw.ipynb', 'repo_name': 'noname30081/IRR', 'size': '2814'}