<jupyter_start><jupyter_text>Package Import<jupyter_code>import urllib.request
import bs4<jupyter_output><empty_output><jupyter_text>Create Request Object for Target url![pageimg](fig1.PNG)<jupyter_code># html object creation
html = urllib.request.urlopen("https://terms.naver.com/list.nhn?cid=51007&categoryId=51007")
print(html)
print(html.read())<jupyter_output>b'<!DOCTYPE html>\n<html lang="ko">\n<head>\n<link rel="canonical" href="https://terms.naver.com/">\n<meta http-equiv="Content-Type" content="text/html; charset=utf-8">\n<meta http-equiv="Content-Script-Type" content="text/javascript">\n<meta http-equiv="Content-Style-Type" content="text/css">\n<meta http-equiv="X-UA-Compatible" content="requiresActiveX=true;IE=edge">\n<meta property="og:type" content="article">\n<meta property="og:url" content="https://terms.naver.com/list.nhn?cid=51007&categoryId=51007">\n<meta property="og:title" content="\xec\x84\x9c\xec\x9a\xb8\xeb\x8c\x80\xed\x95\x99\xea\xb5\x90\xeb\xb3\x91\xec\x9b\x90 \xec\x9d\x98\xed\x95\x99\xec\xa0\x95\xeb\xb3\xb4 : \xeb\x84\xa4\xec\x9d\xb4\xeb\xb2\x84 \xec\xa7\x80\xec\x8b\x9d\xeb\xb0\xb1\xea\xb3\xbc">\n<meta property="og:image" content="https://ssl.pstatic.net/static/terms/common/img/og.png">\n<meta property="og:description" content="\xeb\xa6\xbc\xed\x94\x84\xec\xa2\x85, \xea\xb3\xa0\xec\xa7\x80\xed\x98\x88\xec\xa6\x9d, \xed\[...]<jupyter_text>Create Beautiful Soup Object<jupyter_code>bsObj = bs4.BeautifulSoup(html,"html.parser")<jupyter_output><empty_output><jupyter_text>1. get target based on tags bsObject.find("tag",{"class":"classname"}) - class name is optional- return a corresponding tag (revealed first)![fig2](fig2.PNG)<jupyter_code>firstDiseaseBlock = bsObj.find("div",{"class":"info_area"}) #crawling based on tag and class name
print(firstDiseaseBlock)<jupyter_output><div class="info_area">
<div class="subject">
<strong class="title">
<a href="/entry.nhn?docId=927346&amp;cid=51007&amp;categoryId=51007" onclick="nclk(this, 'tml.termlist', '', '', 1);">림프종 [lymphoma]</a>
<a class="btn_txt __TLIST_PUT" data-contentsid="927346" data-servicecode="TERMS" href="#" onclick="nclk(this, 'tml.kadd','', '', 1)">담기</a>
</strong>
</div>
<p class="desc __ellipsis" style="overflow:hidden;max-height:44px">
					[정의] 림프종은 B-림프구, T-림프구, 자연살해세포(natural killer, NK세포), 리드스텐버그(Reed-Sternberg)세포 또는 림프구와 조직구(팝콘세포)에서 기원하는 림프세포 증식 질환으로 단일 질환이 아니고 이질성 질환의 집합체이다. 비호지킨 림프종은 우리나라 전체 악성 림프종의 95.6%를 차지하며 호지킨 림프종은 4.4%를 차지한다. 비호지킨 림프종 중 미만성 큰 B-세포 림프종(diffuse large B-cell lymphoma)이 41.3%, 결절외변연부세포 림프종(extranodal marginal zone lymphoma of MALT type)이 18.4%, 결절외NK/T-세포 림프종(extranodal NK/T-cell lymphoma, nasal type)이 6.1%, 주변T-세포 림프종(peripheral T-cell lymphoma, unspecified)이 5.8%로 흔하다. 비호지킨 림프종은 온몸에 나타날 수 있으며 종양이 어
				</p>
<div class="related">
<div class="iner">
<span class="info[...]<jupyter_text>2. Category List!["listimage"](./fig3.PNG)- get categories- "class":"content_list"<jupyter_code>ul = bsObj.find("ul",{"class":"content_list"})

print(_list)
_list = ul.findAll("li") # [<li>~~</li>]


for li in _list:
    strong_tag = li.find("strong",{"class":"title"})
    if strong_tag==None :
        continue
    a_tag=strong_tag.find("a")
    print(a_tag.text)<jupyter_output>림프종 [lymphoma]
고지혈증 [hyperlipidemia]
폐렴 [pneumonia]
과민성 대장 증후군 [irritable bowel syndrome]
당뇨병 [diabetes mellitus]
홍역 [measles]
결핵 [Tuberculosis]
크론병 [Crohn’s disease]
천식 [asthma]
급성 신우신염 [acute pyelonephritis]
요로결석 [urinary stone]
두드러기 [hives]
뇌전증 [epilepsy]
편도결석 [tonsillolith]
위궤양 [gastric ulcer]<jupyter_text>Simplified Version<jupyter_code># html object creation
html = urllib.request.urlopen("https://terms.naver.com/list.nhn?cid=51007&categoryId=51007")

ul = bsObj.find("ul",{"class":"content_list"})

_list = ul.findAll("li") # [<li>~~</li>]

for li in _list:
    strong_tag = li.find("strong",{"class":"title"})
    if strong_tag !=None:
        titles.append(strong_tag.find("a").text)

for i in range(len(titles)):
    print(titles[i])<jupyter_output>림프종 [lymphoma]
림프종 [lymphoma]
고지혈증 [hyperlipidemia]
폐렴 [pneumonia]
과민성 대장 증후군 [irritable bowel syndrome]
당뇨병 [diabetes mellitus]
홍역 [measles]
결핵 [Tuberculosis]
크론병 [Crohn’s disease]
천식 [asthma]
급성 신우신염 [acute pyelonephritis]
요로결석 [urinary stone]
두드러기 [hives]
뇌전증 [epilepsy]
편도결석 [tonsillolith]
위궤양 [gastric ulcer]
{'fraction_non_alphanumeric': 0.17841358455454756, 'fraction_numerical': 0.03442661083973017, 'mean_word_length': 6.190635451505017, 'pattern_counts': {'":': 7, '<': 52, '<?xml version=': 0, '>': 50, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '56722', 'license': 'MIT', 'n_tokens_mistral': 2185, 'n_tokens_neox': 2286, 'n_words': 309, 'path': 'BeautifulSoup/.ipynb_checkpoints/BasicCrawling(bs4)-checkpoint.ipynb', 'repo_name': 'minlee077/Basic_Crawling', 'size': '4299'}