<jupyter_start><jupyter_text>mongodb 서버 연결<jupyter_code>client = pymongo.MongoClient("mongodb://172.16.31.10:27017/")
client<jupyter_output><empty_output><jupyter_text>mongodb 컬렉션 생성<jupyter_code>hypebeast = client.crawlingproject.hb
hypebeast
hypebeast_article = hb.Hypebeast()
hypebeast_article
datas = hypebeast_article.one_page(1)
datas.tail(2)<jupyter_output>Crawling : 1 page..
Crawling : 2 page..
Crawling : 3 page..
Crawling : 4 page..
Crawling : 5 page..
Crawling : 6 page..
Crawling : 7 page..
Crawling : 8 page..
Crawling : 9 page..
Crawling : 10 page..
Crawling : 11 page..
Crawling : 12 page..
Crawling : 13 page..
Crawling : 14 page..
Crawling : 15 page..
Crawling : 16 page..
Crawling : 17 page..
Crawling : 18 page..
Crawling : 19 page..<jupyter_text>데이터 프레임 저장<jupyter_code>hb_items = datas.to_dict("records")
hb_items[0]
hb_ids = hypebeast.insert(hb_items)<jupyter_output><empty_output><jupyter_text>데이터 find<jupyter_code>import pandas as pd
datas = hypebeast.find()
QUERY = {"date" : {"$lte" : "Mar2"}}
datas = hypebeast.find(QUERY)
article_df = pd.DataFrame(datas)
article_df<jupyter_output><empty_output><jupyter_text>데이터 전처리<jupyter_code>article_df.drop(columns=["_id"], inplace=True)
article_df[["category", "date", "title", "summary", "content", "link"]]
article_df.sort_values("date", ascending=False).reset_index(drop=True, inplace=True)
article_df
article_df.groupby("category").size()
article_df["category"].unique()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15061058344640435, 'fraction_numerical': 0.03188602442333786, 'mean_word_length': 6.44949494949495, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '57001', 'license': 'MIT', 'n_tokens_mistral': 668, 'n_tokens_neox': 634, 'n_words': 106, 'path': 'Crawling Project_MongoDB_Record.ipynb', 'repo_name': 'saint-hong/CrawlingProject', 'size': '1475'}