<jupyter_start><jupyter_text>Web Scraping  1. Create and send HTTP request to get HTML content I use detik.com to get headline of news<jupyter_code>import requests
website_url = requests.get('https://www.detik.com/').text<jupyter_output><empty_output><jupyter_text>2. use BeautifulSoup from python library to scraping You can install it from pip- pip install BeautifulSoup4Then, if 'lxml' parser getting error. just install it.- pip install lxml<jupyter_code>from bs4 import BeautifulSoup
soup = BeautifulSoup(website_url,'lxml')
print(soup.prettify())<jupyter_output><!DOCTYPE html>
<html lang="id-ID">
 <head>
  <meta content="yes" name="apple-mobile-web-app-capable"/>
  <link href="https://cdn.detik.net.id/detikcom/images/favicon.ico?v=25c7fcf4" rel="shortcut icon" type="image/x-icon"/>
  <title>
   detikcom - Informasi Berita Terupdate Hari Ini
  </title>
  <meta content="Indeks berita terbaru hari ini dari peristiwa, kecelakaan, kriminal, hukum, berita unik, Politik, dan liputan khusus di Indonesia dan Internasional" itemprop="description" name="description"/>
  <meta charset="utf-8"/>
  <meta content="text/html; charset=UTF-8" http-equiv="X-UA-Compatible"/>
  <meta content="width=device-width, initial-scale=1.0" name="viewport"/>
  <link href="https://plus.google.com/+detikcom" rel="publisher"/>
  <meta content="article" property="og:type"/>
  <meta content="detikcom" property="og:site_name"/>
  <meta content="detikcom - Informasi Berita Terupdate Hari Ini" property="og:title"/>
  <meta content="https://cdn.detik.net.id/detik2/images/logo.jpg" [...]<jupyter_text>3. Find the tag of HTML content You can find it from inspect the element of website too<jupyter_code>My_table = soup.find('div',{'class': 'list-content'})
My_table<jupyter_output><empty_output><jupyter_text>this is to find headline from 'a' tag HTML.You can find something else.<jupyter_code>links = My_table.findAll('a')
links<jupyter_output><empty_output><jupyter_text>4. Looping the result and get into list I use "link.findAll(text=True)" to get an string inside tag. Then, substring to string that I want.<jupyter_code>news = []
for link in links:
    if link.findAll(text=True) != ['\n', '\n', ' ', '\n']:
        colleges.append(link.findAll(text=True)[0][2:-1:])
print(news)<jupyter_output>['Jokowi: Per 12 Maret, 2 Pasien Corona Meninggal di Indonesia', 'Kantor Terawan Dapat Tambahan Rp 1 Triliun dari Sri Mulyani', 'Asyik! Pre-order Vivo V19 Dapat Cashback Rp 300.000', 'Wow! BRI Buyback Saham Rp 3 T', 'Balada Maskapai RI: Turis Asing Tiarap, Domestiknya Nggak Liburan']<jupyter_text>5. Convert into Dataframe (Optional)<jupyter_code>import pandas as pd
df = pd.DataFrame()
df['berita'] = news
df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13688073394495412, 'fraction_numerical': 0.011743119266055046, 'mean_word_length': 5.832080200501253, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 38, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 3}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '531849', 'license': 'MIT', 'n_tokens_mistral': 989, 'n_tokens_neox': 955, 'n_words': 252, 'path': 'web_scraping.ipynb', 'repo_name': 'adi6ntro/simple-web-scraping', 'size': '2725'}